2024-03-18T14:29:51.299 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T14:29:51.299 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} --name pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831 --owner scheduled_pdonnell@teuthology --verbose -- /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/orig.config.yaml 2024-03-18T14:29:51.343 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T14:29:51.441 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831 branch: wip-batrick-testing-20240314.183243 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} email: pdonnell@redhat.com first_in_suite: false job_id: '7608831' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - out of quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQtYpzYGP9RtuDnRfRAjMIFbIwY8l0IVJ419joOtQa91wnFCza3eQSAAxQj5WQX7MEHuvT3LFPqwb5o4+jEM8M= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1btTUr4MtuN4TIKkNYtJ8gv15N6zwZK5VqgoONnMI+bye+UznISZRSsJ0R/IXGCamLgfoAwgbJ30P0fFhjZ3g= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_14:19:46 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T14:29:51.441 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa; will attempt to use it 2024-03-18T14:29:51.442 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks 2024-03-18T14:29:51.442 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T14:29:51.444 INFO:teuthology.task.internal:Checking packages... 2024-03-18T14:29:51.465 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T14:29:51.465 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T14:29:51.465 INFO:teuthology.packaging:ref: None 2024-03-18T14:29:51.465 INFO:teuthology.packaging:tag: None 2024-03-18T14:29:51.466 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T14:29:51.466 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:29:51.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T14:29:51.678 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T14:29:51.680 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T14:29:51.687 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T14:29:51.687 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T14:29:51.703 INFO:teuthology.task.internal:Saving configuration 2024-03-18T14:29:51.718 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T14:29:51.724 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T14:29:51.750 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:18.842886', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQtYpzYGP9RtuDnRfRAjMIFbIwY8l0IVJ419joOtQa91wnFCza3eQSAAxQj5WQX7MEHuvT3LFPqwb5o4+jEM8M='} 2024-03-18T14:29:51.769 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:18.840893', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1btTUr4MtuN4TIKkNYtJ8gv15N6zwZK5VqgoONnMI+bye+UznISZRSsJ0R/IXGCamLgfoAwgbJ30P0fFhjZ3g='} 2024-03-18T14:29:51.769 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T14:29:51.776 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-18T14:29:51.776 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-18T14:29:51.776 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T14:29:51.849 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4178e3f160>, signals=[15]) 2024-03-18T14:29:51.849 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T14:29:51.855 INFO:teuthology.task.internal:Opening connections... 2024-03-18T14:29:51.855 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:29:51.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:29:51.965 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:29:51.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:29:52.056 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T14:29:52.062 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-03-18T14:29:52.080 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-03-18T14:29:52.081 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-03-18T14:29:52.138 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:29:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:29:52.140 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-03-18T14:29:52.165 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-03-18T14:29:52.182 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-03-18T14:29:52.183 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-03-18T14:29:52.238 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-03-18T14:29:52.238 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-03-18T14:29:52.238 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-03-18T14:29:52.238 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:29:52.238 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-03-18T14:29:52.238 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:29:52.239 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:29:52.239 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-03-18T14:29:52.257 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T14:29:52.266 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T14:29:52.271 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T14:29:52.271 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:29:52.273 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:29:52.293 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T14:29:52.300 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T14:29:52.300 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:29:52.331 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:29:52.358 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T14:29:52.413 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T14:29:52.426 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-18T14:29:52.426 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-18T14:29:52.426 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:29:52.427 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:29:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:29:52.442 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-03-18T14:29:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:29:52.443 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-18T14:29:52.497 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-361.el9.x86_64 2024-03-18T14:29:52.497 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-361.el9.x86_64 2024-03-18T14:29:52.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2024-03-18T14:29:52.501 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-03-18T14:29:52.502 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-361.el9.x86_64 2024-03-18T14:29:52.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2024-03-18T14:29:54.649 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-18T14:29:54.725 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-18T14:29:56.025 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-18T14:29:56.176 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-18T14:29:57.349 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.0 MB 00:00 2024-03-18T14:29:58.598 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-18T14:29:59.492 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-18T14:30:00.449 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-18T14:30:04.525 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-18T14:30:04.726 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-18T14:30:06.443 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 16 kB 00:00 2024-03-18T14:30:07.210 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-18T14:30:08.270 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-18T14:30:09.918 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-03-18T14:30:13.379 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-18T14:30:14.843 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-18T14:30:15.685 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T14:30:15.686 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T14:30:15.774 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-18T14:30:15.780 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-03-18T14:30:15.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:30:15.782 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 100 M 2024-03-18T14:30:15.782 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 123 M 2024-03-18T14:30:15.782 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-18T14:30:16.388 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-18T14:30:16.722 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 33 MB/s | 21 MB 00:00 2024-03-18T14:30:17.058 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T14:30:17.058 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T14:30:17.114 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 46 MB/s | 33 MB 00:00 2024-03-18T14:30:17.167 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-18T14:30:17.179 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-18T14:30:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-03-18T14:30:17.179 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-18T14:30:17.179 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-18T14:30:17.179 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2024-03-18T14:30:17.180 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:30:17.181 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 100 M 2024-03-18T14:30:17.181 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 123 M 2024-03-18T14:30:17.181 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-18T14:30:17.564 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 27 MB/s | 39 MB 00:01 2024-03-18T14:30:17.564 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:30:17.565 INFO:teuthology.orchestra.run.smithi006.stdout:Total 56 MB/s | 100 MB 00:01 2024-03-18T14:30:18.484 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-18T14:30:18.619 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 5.2 MB/s | 6.0 MB 00:01 2024-03-18T14:30:18.906 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-18T14:30:18.907 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-18T14:30:19.061 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 13 MB/s | 21 MB 00:01 2024-03-18T14:30:20.337 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-18T14:30:20.337 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-18T14:30:20.994 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 11 MB/s | 39 MB 00:03 2024-03-18T14:30:23.179 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-18T14:30:23.905 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-18T14:30:24.040 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:30:24.743 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 5.5 MB/s | 33 MB 00:06 2024-03-18T14:30:24.744 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:30:24.744 INFO:teuthology.orchestra.run.smithi092.stdout:Total 13 MB/s | 100 MB 00:07 2024-03-18T14:30:25.633 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-18T14:30:25.998 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-18T14:30:25.999 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-18T14:30:26.946 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:30:27.159 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:30:27.336 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-18T14:30:27.336 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-18T14:30:30.024 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-18T14:30:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-18T14:30:30.793 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:30:32.256 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:30:32.382 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:33.605 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:30:33.788 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:30:37.509 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:38.534 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:30:38.664 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:30:43.398 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:08.455 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:08.455 INFO:teuthology.orchestra.run.smithi006.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T14:31:08.455 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:31:08.542 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:08.973 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:08.973 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-18T14:31:08.973 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:31:08.973 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:31:09.433 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:31:09.434 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-18T14:31:09.958 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T14:31:10.151 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T14:31:10.151 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T14:31:10.152 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-18T14:31:10.152 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T14:31:10.152 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T14:31:10.152 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-18T14:31:10.152 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T14:31:10.152 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T14:31:10.278 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T14:31:10.278 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T14:31:10.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T14:31:10.324 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:31:10.325 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T14:31:10.326 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T14:31:10.326 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-427.el9.x86_64 2024-03-18T14:31:10.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T14:31:10.959 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-03-18T14:31:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T14:31:12.561 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-03-18T14:31:12.564 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-18T14:31:12.580 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.WS9ngYWRS2 2024-03-18T14:31:12.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WS9ngYWRS2 2024-03-18T14:31:12.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.WS9ngYWRS2 2024-03-18T14:31:12.784 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.WS9ngYWRS2 is 6KB 2024-03-18T14:31:12.832 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.WS9ngYWRS2 2024-03-18T14:31:12.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T14:31:12.895 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:12.895 INFO:teuthology.orchestra.run.smithi092.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T14:31:12.896 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:31:12.915 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T14:31:12.915 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T14:31:12.915 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T14:31:12.915 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-18T14:31:12.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-18T14:31:12.980 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:13.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-03-18T14:31:13.463 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:13.464 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-18T14:31:13.464 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T14:31:13.464 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T14:31:13.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T14:31:13.892 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:31:13.892 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-18T14:31:13.892 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-18T14:31:13.892 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-18T14:31:13.892 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-18T14:31:13.892 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-18T14:31:13.893 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:31:13.893 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-18T14:31:14.403 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T14:31:14.550 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T14:31:14.550 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T14:31:14.550 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-18T14:31:14.550 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T14:31:14.550 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T14:31:14.551 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-18T14:31:14.551 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T14:31:14.551 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T14:31:14.675 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T14:31:14.675 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T14:31:14.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2024-03-18T14:31:14.713 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2024-03-18T14:31:14.713 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2024-03-18T14:31:14.714 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2024-03-18T14:31:14.714 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2024-03-18T14:31:14.714 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2024-03-18T14:31:14.714 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T14:31:14.714 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2024-03-18T14:31:14.714 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T14:31:14.715 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:31:14.716 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T14:31:14.718 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T14:31:14.718 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-427.el9.x86_64 2024-03-18T14:31:14.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T14:31:15.329 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-03-18T14:31:16.851 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T14:31:16.890 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-03-18T14:31:16.892 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-18T14:31:16.948 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.fKDwCpAW5j 2024-03-18T14:31:16.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fKDwCpAW5j 2024-03-18T14:31:17.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.fKDwCpAW5j 2024-03-18T14:31:17.168 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.fKDwCpAW5j is 6KB 2024-03-18T14:31:17.217 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.fKDwCpAW5j 2024-03-18T14:31:17.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T14:31:17.295 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T14:31:17.295 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T14:31:17.296 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T14:31:17.296 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-18T14:31:17.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-18T14:31:17.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-03-18T14:31:43.106 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-18T14:31:43.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-18T14:31:43.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:31:47.454 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-18T14:31:47.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-18T14:31:47.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:32:01.571 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-03-18T14:32:05.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-03-18T14:32:10.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-18T14:32:10.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:32:14.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-18T14:32:14.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:32:17.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-03-18T14:32:29.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-18T14:32:29.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:32:48.419 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-03-18T14:33:03.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-18T14:33:03.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:33:03.792 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-03-18T14:33:04.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-18T14:33:04.236 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-18T14:33:04.236 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-03-18T14:33:04.250 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-427.el9.x86_64 2024-03-18T14:33:04.250 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-18T14:33:04.250 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T14:33:04.250 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T14:33:05.251 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-18T14:33:05.252 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T14:33:05.336 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2024-03-18T14:33:05.370 DEBUG:teuthology.parallel:result is None 2024-03-18T14:33:10.580 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T14:33:22.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-18T14:33:22.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:33:22.867 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-18T14:33:23.259 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-18T14:33:23.259 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-18T14:33:23.259 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-18T14:33:23.274 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-427.el9.x86_64 2024-03-18T14:33:23.275 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-18T14:33:23.275 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T14:33:23.275 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T14:33:24.276 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-18T14:33:24.276 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T14:33:24.351 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-03-18T14:33:24.386 DEBUG:teuthology.parallel:result is None 2024-03-18T14:33:24.387 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T14:33:24.394 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T14:33:24.394 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:33:24.397 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:33:24.415 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T14:33:24.422 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T14:33:24.431 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T14:33:24.431 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:33:24.454 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:33:24.631 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T14:33:24.642 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T14:33:24.643 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:33:24.645 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:33:24.678 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:24.680 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:24.692 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:24.694 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:33:24.699 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T14:33:24.707 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T14:33:24.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:33:24.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:33:24.771 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T14:33:24.783 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T14:33:24.783 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:33:24.811 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:33:24.827 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:24.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:24.956 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:25.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:25.081 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:33:25.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:33:25.149 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:25.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:33:25.244 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:25.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:33:25.370 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:33:25.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:33:25.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-03-18T14:33:25.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-03-18T14:33:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:33:25.547 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:33:25.871 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T14:33:25.879 INFO:teuthology.task.internal:Starting timer... 2024-03-18T14:33:25.879 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T14:33:25.903 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T14:33:25.913 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-18T14:33:25.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-03-18T14:33:25.974 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-03-18T14:33:25.976 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-03-18T14:33:26.045 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-03-18T14:33:26.047 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T14:33:26.047 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-03-18T14:33:26.073 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-03-18T14:33:26.074 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-03-18T14:33:26.104 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-03-18T14:33:26.104 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive'} 2024-03-18T14:33:26.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:33:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:33:26.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:33:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:33:26.171 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T14:33:26.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:33:26.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:33:26.247 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T14:33:26.258 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T14:33:26.278 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T14:33:26.280 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi092.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T14:38:09.582 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com')] 2024-03-18T14:38:09.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-18T14:38:09.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:38:09.681 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-18T14:38:09.761 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-18T14:38:09.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-18T14:38:09.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:38:09.852 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-03-18T14:38:09.932 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-18T14:38:09.932 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T14:38:09.944 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T14:38:09.944 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T14:38:09.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T14:38:09.947 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T14:38:09.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T14:38:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T14:38:09.994 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T14:38:10.003 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T14:38:10.019 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T14:38:10.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-03-18T14:38:10.038 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-03-18T14:38:10.048 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-03-18T14:38:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T14:38:10.062 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-03-18T14:38:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T14:38:10.078 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T14:38:10.095 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T14:38:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-18T14:38:10.127 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T14:38:10.127 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-18T14:38:10.127 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.127 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.127 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.127 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.146 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-18T14:38:10.150 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T14:38:10.151 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-18T14:38:10.151 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.151 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.151 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.151 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:38:10.151 INFO:teuthology.run_tasks:Running task install... 2024-03-18T14:38:10.161 DEBUG:teuthology.task.install:project ceph 2024-03-18T14:38:10.162 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T14:38:10.162 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T14:38:10.162 INFO:teuthology.task.install:Using flavor: default 2024-03-18T14:38:10.171 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T14:38:10.171 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T14:38:10.171 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T14:38:10.172 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:38:10.174 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T14:38:10.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:38:10.321 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T14:38:10.321 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T14:38:10.324 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T14:38:10.325 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T14:38:10.549 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T14:38:10.550 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:38:10.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T14:38:10.579 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T14:38:10.580 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T14:38:10.602 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T14:38:10.602 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:38:10.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T14:38:10.630 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T14:38:10.630 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T14:38:10.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T14:38:10.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T14:38:10.728 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T14:38:10.779 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T14:38:10.794 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-03-18T14:38:10.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-03-18T14:38:10.845 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2024-03-18T14:38:10.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2024-03-18T14:38:11.045 INFO:teuthology.orchestra.run.smithi006.stdout:74 files removed 2024-03-18T14:38:11.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T14:38:11.086 INFO:teuthology.orchestra.run.smithi092.stdout:74 files removed 2024-03-18T14:38:11.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T14:38:11.743 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 238 kB/s | 83 kB 00:00 2024-03-18T14:38:12.244 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-18T14:38:12.532 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 75 kB/s | 83 kB 00:01 2024-03-18T14:38:12.694 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-18T14:38:12.857 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 37 kB/s | 10 kB 00:00 2024-03-18T14:38:13.458 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-03-18T14:38:14.252 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 8.8 MB/s | 6.7 MB 00:00 2024-03-18T14:38:14.864 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 3.1 MB/s | 6.7 MB 00:02 2024-03-18T14:38:15.794 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-18T14:38:16.759 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-18T14:38:17.423 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 1.4 MB/s | 2.0 MB 00:01 2024-03-18T14:38:18.421 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-18T14:38:19.027 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-18T14:38:21.630 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 19 MB 00:02 2024-03-18T14:38:25.867 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-18T14:38:27.351 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-03-18T14:38:28.227 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-18T14:38:29.911 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-18T14:38:31.713 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-18T14:38:35.660 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 2.5 MB/s | 21 MB 00:08 2024-03-18T14:38:36.687 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-18T14:38:39.226 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:39.227 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:39.242 INFO:teuthology.orchestra.run.smithi092.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T14:38:39.243 INFO:teuthology.orchestra.run.smithi092.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T14:38:39.244 INFO:teuthology.orchestra.run.smithi092.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T14:38:39.245 INFO:teuthology.orchestra.run.smithi092.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:39.249 INFO:teuthology.orchestra.run.smithi092.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:39.250 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T14:38:39.251 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T14:38:39.332 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-18T14:38:39.355 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-18T14:38:39.355 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-18T14:38:39.355 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-18T14:38:39.355 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-18T14:38:39.355 INFO:teuthology.orchestra.run.smithi092.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T14:38:39.355 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T14:38:39.355 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T14:38:39.356 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T14:38:39.357 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T14:38:39.358 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T14:38:39.359 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T14:38:39.359 INFO:teuthology.orchestra.run.smithi092.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T14:38:39.359 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T14:38:39.359 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T14:38:39.359 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T14:38:39.359 INFO:teuthology.orchestra.run.smithi092.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T14:38:39.359 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T14:38:39.360 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T14:38:39.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T14:38:39.362 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T14:38:39.363 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T14:38:39.364 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T14:38:39.365 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T14:38:39.366 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T14:38:39.366 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T14:38:39.366 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T14:38:39.366 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T14:38:39.366 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T14:38:39.366 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T14:38:39.366 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T14:38:39.367 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T14:38:39.368 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T14:38:39.369 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T14:38:39.370 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout:Install 150 Packages 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 24 Packages 2024-03-18T14:38:39.371 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:38:39.372 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 224 M 2024-03-18T14:38:39.372 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-18T14:38:40.755 INFO:teuthology.orchestra.run.smithi092.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-18T14:38:40.919 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-18T14:38:41.114 INFO:teuthology.orchestra.run.smithi092.stdout:(2/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 7.8 MB/s | 5.4 MB 00:00 2024-03-18T14:38:41.164 INFO:teuthology.orchestra.run.smithi092.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.8 MB/s | 145 kB 00:00 2024-03-18T14:38:41.356 INFO:teuthology.orchestra.run.smithi092.stdout:(4/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-18T14:38:41.406 INFO:teuthology.orchestra.run.smithi092.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 9.8 MB/s | 2.4 MB 00:00 2024-03-18T14:38:41.765 INFO:teuthology.orchestra.run.smithi092.stdout:(6/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 13 MB/s | 4.7 MB 00:00 2024-03-18T14:38:41.965 INFO:teuthology.orchestra.run.smithi092.stdout:(7/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 2.5 MB/s | 1.5 MB 00:00 2024-03-18T14:38:42.757 INFO:teuthology.orchestra.run.smithi092.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 17 MB/s | 17 MB 00:00 2024-03-18T14:38:42.790 INFO:teuthology.orchestra.run.smithi092.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 760 kB/s | 25 kB 00:00 2024-03-18T14:38:43.531 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:43.532 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T14:38:43.545 INFO:teuthology.orchestra.run.smithi006.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T14:38:43.546 INFO:teuthology.orchestra.run.smithi006.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T14:38:43.547 INFO:teuthology.orchestra.run.smithi006.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T14:38:43.548 INFO:teuthology.orchestra.run.smithi006.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:43.549 INFO:teuthology.orchestra.run.smithi006.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T14:38:43.552 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T14:38:43.553 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T14:38:43.640 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-18T14:38:43.665 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-18T14:38:43.665 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-18T14:38:43.665 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-18T14:38:43.665 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-18T14:38:43.665 INFO:teuthology.orchestra.run.smithi006.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T14:38:43.665 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T14:38:43.666 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T14:38:43.667 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T14:38:43.668 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T14:38:43.669 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T14:38:43.670 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T14:38:43.671 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T14:38:43.672 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T14:38:43.673 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T14:38:43.674 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T14:38:43.675 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T14:38:43.676 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T14:38:43.677 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-03-18T14:38:43.678 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T14:38:43.679 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:38:43.679 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-18T14:38:43.679 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-18T14:38:43.679 INFO:teuthology.orchestra.run.smithi006.stdout:Install 150 Packages 2024-03-18T14:38:43.679 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 24 Packages 2024-03-18T14:38:43.679 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:38:43.680 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 224 M 2024-03-18T14:38:43.680 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-18T14:38:44.753 INFO:teuthology.orchestra.run.smithi006.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-18T14:38:45.428 INFO:teuthology.orchestra.run.smithi006.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 890 kB 00:00 2024-03-18T14:38:45.529 INFO:teuthology.orchestra.run.smithi006.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.4 MB/s | 145 kB 00:00 2024-03-18T14:38:45.940 INFO:teuthology.orchestra.run.smithi092.stdout:(10/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 15 MB/s | 48 MB 00:03 2024-03-18T14:38:46.032 INFO:teuthology.orchestra.run.smithi092.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 221 kB 00:00 2024-03-18T14:38:46.116 INFO:teuthology.orchestra.run.smithi092.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 325 kB/s | 27 kB 00:00 2024-03-18T14:38:46.199 INFO:teuthology.orchestra.run.smithi092.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 381 kB/s | 32 kB 00:00 2024-03-18T14:38:46.312 INFO:teuthology.orchestra.run.smithi006.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-03-18T14:38:46.350 INFO:teuthology.orchestra.run.smithi092.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.9 MB/s | 747 kB 00:00 2024-03-18T14:38:46.363 INFO:teuthology.orchestra.run.smithi006.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 2.8 MB/s | 2.4 MB 00:00 2024-03-18T14:38:46.425 INFO:teuthology.orchestra.run.smithi092.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-18T14:38:46.517 INFO:teuthology.orchestra.run.smithi092.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 127 kB 00:00 2024-03-18T14:38:46.747 INFO:teuthology.orchestra.run.smithi006.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-03-18T14:38:46.934 INFO:teuthology.orchestra.run.smithi092.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 1.2 MB/s | 501 kB 00:00 2024-03-18T14:38:47.513 INFO:teuthology.orchestra.run.smithi006.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 4.1 MB/s | 4.7 MB 00:01 2024-03-18T14:38:47.601 INFO:teuthology.orchestra.run.smithi092.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 7.7 MB/s | 5.1 MB 00:00 2024-03-18T14:38:47.684 INFO:teuthology.orchestra.run.smithi092.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 542 kB/s | 45 kB 00:00 2024-03-18T14:38:47.768 INFO:teuthology.orchestra.run.smithi092.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-18T14:38:47.852 INFO:teuthology.orchestra.run.smithi092.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-18T14:38:47.944 INFO:teuthology.orchestra.run.smithi092.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.4 MB/s | 320 kB 00:00 2024-03-18T14:38:48.044 INFO:teuthology.orchestra.run.smithi092.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.9 MB/s | 298 kB 00:00 2024-03-18T14:38:48.127 INFO:teuthology.orchestra.run.smithi092.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 99 kB 00:00 2024-03-18T14:38:48.211 INFO:teuthology.orchestra.run.smithi092.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-18T14:38:48.736 INFO:teuthology.orchestra.run.smithi092.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 5.9 MB/s | 3.1 MB 00:00 2024-03-18T14:38:48.829 INFO:teuthology.orchestra.run.smithi092.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-18T14:38:48.912 INFO:teuthology.orchestra.run.smithi092.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 407 kB/s | 34 kB 00:00 2024-03-18T14:38:49.012 INFO:teuthology.orchestra.run.smithi092.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.4 MB/s | 139 kB 00:00 2024-03-18T14:38:49.438 INFO:teuthology.orchestra.run.smithi092.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 4.2 MB/s | 1.8 MB 00:00 2024-03-18T14:38:50.280 INFO:teuthology.orchestra.run.smithi006.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 4.8 MB/s | 17 MB 00:03 2024-03-18T14:38:50.372 INFO:teuthology.orchestra.run.smithi006.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 276 kB/s | 25 kB 00:00 2024-03-18T14:38:50.730 INFO:teuthology.orchestra.run.smithi092.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 5.7 MB/s | 7.4 MB 00:01 2024-03-18T14:38:50.838 INFO:teuthology.orchestra.run.smithi092.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.2 MB/s | 248 kB 00:00 2024-03-18T14:38:50.914 INFO:teuthology.orchestra.run.smithi092.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 657 kB/s | 49 kB 00:00 2024-03-18T14:38:50.989 INFO:teuthology.orchestra.run.smithi092.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 219 kB/s | 16 kB 00:00 2024-03-18T14:38:51.089 INFO:teuthology.orchestra.run.smithi092.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.6 MB/s | 264 kB 00:00 2024-03-18T14:38:51.256 INFO:teuthology.orchestra.run.smithi092.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-18T14:38:51.323 INFO:teuthology.orchestra.run.smithi092.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 274 kB/s | 18 kB 00:00 2024-03-18T14:38:51.381 INFO:teuthology.orchestra.run.smithi006.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 2.5 MB/s | 9.8 MB 00:03 2024-03-18T14:38:51.497 INFO:teuthology.orchestra.run.smithi006.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.9 MB/s | 221 kB 00:00 2024-03-18T14:38:51.557 INFO:teuthology.orchestra.run.smithi092.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2024-03-18T14:38:51.572 INFO:teuthology.orchestra.run.smithi006.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 361 kB/s | 27 kB 00:00 2024-03-18T14:38:51.648 INFO:teuthology.orchestra.run.smithi092.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 9.8 MB/s | 920 kB 00:00 2024-03-18T14:38:51.656 INFO:teuthology.orchestra.run.smithi006.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 380 kB/s | 32 kB 00:00 2024-03-18T14:38:51.690 INFO:teuthology.orchestra.run.smithi092.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-18T14:38:51.733 INFO:teuthology.orchestra.run.smithi092.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-18T14:38:51.775 INFO:teuthology.orchestra.run.smithi092.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 761 kB/s | 32 kB 00:00 2024-03-18T14:38:52.015 INFO:teuthology.orchestra.run.smithi006.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 2.0 MB/s | 747 kB 00:00 2024-03-18T14:38:52.140 INFO:teuthology.orchestra.run.smithi006.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.3 MB/s | 163 kB 00:00 2024-03-18T14:38:52.142 INFO:teuthology.orchestra.run.smithi092.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.1 MB/s | 3.0 MB 00:00 2024-03-18T14:38:52.192 INFO:teuthology.orchestra.run.smithi092.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 343 kB/s | 17 kB 00:00 2024-03-18T14:38:52.207 INFO:teuthology.orchestra.run.smithi006.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-18T14:38:52.226 INFO:teuthology.orchestra.run.smithi092.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-18T14:38:52.268 INFO:teuthology.orchestra.run.smithi092.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-18T14:38:52.301 INFO:teuthology.orchestra.run.smithi092.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T14:38:52.360 INFO:teuthology.orchestra.run.smithi092.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-18T14:38:52.418 INFO:teuthology.orchestra.run.smithi092.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 9.7 MB/s | 578 kB 00:00 2024-03-18T14:38:52.449 INFO:teuthology.orchestra.run.smithi006.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.0 MB/s | 501 kB 00:00 2024-03-18T14:38:52.460 INFO:teuthology.orchestra.run.smithi092.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-18T14:38:52.519 INFO:teuthology.orchestra.run.smithi092.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-18T14:38:52.561 INFO:teuthology.orchestra.run.smithi092.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-18T14:38:52.628 INFO:teuthology.orchestra.run.smithi092.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-18T14:38:52.662 INFO:teuthology.orchestra.run.smithi092.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-18T14:38:52.820 INFO:teuthology.orchestra.run.smithi092.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-18T14:38:52.887 INFO:teuthology.orchestra.run.smithi092.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-18T14:38:52.946 INFO:teuthology.orchestra.run.smithi092.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-18T14:38:52.988 INFO:teuthology.orchestra.run.smithi092.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-18T14:38:53.172 INFO:teuthology.orchestra.run.smithi092.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-18T14:38:53.230 INFO:teuthology.orchestra.run.smithi092.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-18T14:38:53.264 INFO:teuthology.orchestra.run.smithi092.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-18T14:38:53.439 INFO:teuthology.orchestra.run.smithi092.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-18T14:38:53.465 INFO:teuthology.orchestra.run.smithi092.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 977 kB/s | 24 kB 00:00 2024-03-18T14:38:53.499 INFO:teuthology.orchestra.run.smithi092.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 543 kB/s | 18 kB 00:00 2024-03-18T14:38:53.532 INFO:teuthology.orchestra.run.smithi092.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-18T14:38:53.557 INFO:teuthology.orchestra.run.smithi092.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 1.2 MB/s | 32 kB 00:00 2024-03-18T14:38:53.782 INFO:teuthology.orchestra.run.smithi092.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-18T14:38:53.849 INFO:teuthology.orchestra.run.smithi092.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.2 MB/s | 217 kB 00:00 2024-03-18T14:38:53.908 INFO:teuthology.orchestra.run.smithi092.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-18T14:38:53.950 INFO:teuthology.orchestra.run.smithi092.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-18T14:38:54.058 INFO:teuthology.orchestra.run.smithi092.stdout:(71/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 832 kB/s | 9.8 MB 00:12 2024-03-18T14:38:54.084 INFO:teuthology.orchestra.run.smithi092.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 307 kB/s | 41 kB 00:00 2024-03-18T14:38:54.109 INFO:teuthology.orchestra.run.smithi092.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-18T14:38:54.134 INFO:teuthology.orchestra.run.smithi092.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T14:38:54.160 INFO:teuthology.orchestra.run.smithi092.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T14:38:54.194 INFO:teuthology.orchestra.run.smithi092.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 809 kB/s | 47 kB 00:00 2024-03-18T14:38:54.219 INFO:teuthology.orchestra.run.smithi092.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 729 kB/s | 43 kB 00:00 2024-03-18T14:38:54.332 INFO:teuthology.orchestra.run.smithi006.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 2.7 MB/s | 5.1 MB 00:01 2024-03-18T14:38:54.402 INFO:teuthology.orchestra.run.smithi092.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 212 kB/s | 39 kB 00:00 2024-03-18T14:38:54.424 INFO:teuthology.orchestra.run.smithi006.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 493 kB/s | 45 kB 00:00 2024-03-18T14:38:54.645 INFO:teuthology.orchestra.run.smithi092.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-18T14:38:54.669 INFO:teuthology.orchestra.run.smithi092.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 155 kB/s | 41 kB 00:00 2024-03-18T14:38:54.695 INFO:teuthology.orchestra.run.smithi092.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-18T14:38:54.720 INFO:teuthology.orchestra.run.smithi092.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T14:38:54.745 INFO:teuthology.orchestra.run.smithi092.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T14:38:54.771 INFO:teuthology.orchestra.run.smithi092.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-18T14:38:54.796 INFO:teuthology.orchestra.run.smithi092.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-18T14:38:54.838 INFO:teuthology.orchestra.run.smithi092.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-18T14:38:54.880 INFO:teuthology.orchestra.run.smithi092.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-18T14:38:54.905 INFO:teuthology.orchestra.run.smithi092.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-18T14:38:54.931 INFO:teuthology.orchestra.run.smithi092.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T14:38:54.956 INFO:teuthology.orchestra.run.smithi092.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-18T14:38:55.098 INFO:teuthology.orchestra.run.smithi092.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-18T14:38:55.149 INFO:teuthology.orchestra.run.smithi092.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-18T14:38:55.174 INFO:teuthology.orchestra.run.smithi092.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-03-18T14:38:55.199 INFO:teuthology.orchestra.run.smithi092.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-18T14:38:55.216 INFO:teuthology.orchestra.run.smithi092.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 514 kB/s | 21 kB 00:00 2024-03-18T14:38:55.233 INFO:teuthology.orchestra.run.smithi092.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 758 kB/s | 25 kB 00:00 2024-03-18T14:38:55.258 INFO:teuthology.orchestra.run.smithi006.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 157 kB/s | 131 kB 00:00 2024-03-18T14:38:55.283 INFO:teuthology.orchestra.run.smithi092.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-03-18T14:38:55.334 INFO:teuthology.orchestra.run.smithi092.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-03-18T14:38:55.401 INFO:teuthology.orchestra.run.smithi092.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 22 MB/s | 4.1 MB 00:00 2024-03-18T14:38:55.425 INFO:teuthology.orchestra.run.smithi006.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 972 kB/s | 162 kB 00:00 2024-03-18T14:38:55.426 INFO:teuthology.orchestra.run.smithi092.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-18T14:38:55.451 INFO:teuthology.orchestra.run.smithi092.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 696 kB/s | 35 kB 00:00 2024-03-18T14:38:55.476 INFO:teuthology.orchestra.run.smithi092.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-18T14:38:55.502 INFO:teuthology.orchestra.run.smithi092.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-18T14:38:55.560 INFO:teuthology.orchestra.run.smithi092.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 266 kB/s | 22 kB 00:00 2024-03-18T14:38:55.667 INFO:teuthology.orchestra.run.smithi006.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 1.3 MB/s | 320 kB 00:00 2024-03-18T14:38:55.677 INFO:teuthology.orchestra.run.smithi092.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-03-18T14:38:55.892 INFO:teuthology.orchestra.run.smithi006.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 298 kB 00:00 2024-03-18T14:38:56.001 INFO:teuthology.orchestra.run.smithi006.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 919 kB/s | 99 kB 00:00 2024-03-18T14:38:56.101 INFO:teuthology.orchestra.run.smithi006.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 853 kB/s | 85 kB 00:00 2024-03-18T14:38:56.303 INFO:teuthology.orchestra.run.smithi092.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 739 kB/s | 548 kB 00:00 2024-03-18T14:38:56.328 INFO:teuthology.orchestra.run.smithi092.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 475 kB/s | 309 kB 00:00 2024-03-18T14:38:56.420 INFO:teuthology.orchestra.run.smithi092.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-18T14:38:56.554 INFO:teuthology.orchestra.run.smithi092.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-03-18T14:38:56.679 INFO:teuthology.orchestra.run.smithi092.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-03-18T14:38:56.846 INFO:teuthology.orchestra.run.smithi092.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2024-03-18T14:38:56.871 INFO:teuthology.orchestra.run.smithi092.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-03-18T14:38:56.997 INFO:teuthology.orchestra.run.smithi092.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-18T14:38:57.147 INFO:teuthology.orchestra.run.smithi092.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-18T14:38:57.180 INFO:teuthology.orchestra.run.smithi092.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 2.9 MB/s | 548 kB 00:00 2024-03-18T14:38:57.256 INFO:teuthology.orchestra.run.smithi092.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-03-18T14:38:57.281 INFO:teuthology.orchestra.run.smithi092.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-18T14:38:57.381 INFO:teuthology.orchestra.run.smithi092.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 119 kB/s | 15 kB 00:00 2024-03-18T14:38:57.406 INFO:teuthology.orchestra.run.smithi092.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-03-18T14:38:57.523 INFO:teuthology.orchestra.run.smithi092.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-18T14:38:57.549 INFO:teuthology.orchestra.run.smithi092.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-03-18T14:38:57.616 INFO:teuthology.orchestra.run.smithi092.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-03-18T14:38:57.683 INFO:teuthology.orchestra.run.smithi092.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-18T14:38:57.792 INFO:teuthology.orchestra.run.smithi092.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-18T14:38:57.817 INFO:teuthology.orchestra.run.smithi092.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-18T14:38:57.909 INFO:teuthology.orchestra.run.smithi092.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-03-18T14:38:57.926 INFO:teuthology.orchestra.run.smithi092.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-03-18T14:38:58.001 INFO:teuthology.orchestra.run.smithi092.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-03-18T14:38:58.018 INFO:teuthology.orchestra.run.smithi092.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-18T14:38:58.093 INFO:teuthology.orchestra.run.smithi092.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-18T14:38:58.176 INFO:teuthology.orchestra.run.smithi006.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 1.5 MB/s | 3.1 MB 00:02 2024-03-18T14:38:58.185 INFO:teuthology.orchestra.run.smithi092.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-03-18T14:38:58.335 INFO:teuthology.orchestra.run.smithi006.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.1 MB/s | 170 kB 00:00 2024-03-18T14:38:58.418 INFO:teuthology.orchestra.run.smithi006.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 409 kB/s | 34 kB 00:00 2024-03-18T14:38:58.427 INFO:teuthology.orchestra.run.smithi092.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 2.5 MB/s | 1.0 MB 00:00 2024-03-18T14:38:58.452 INFO:teuthology.orchestra.run.smithi092.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 295 kB/s | 79 kB 00:00 2024-03-18T14:38:58.569 INFO:teuthology.orchestra.run.smithi006.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 924 kB/s | 139 kB 00:00 2024-03-18T14:38:58.569 INFO:teuthology.orchestra.run.smithi092.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 407 kB/s | 58 kB 00:00 2024-03-18T14:38:58.645 INFO:teuthology.orchestra.run.smithi092.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-18T14:38:58.778 INFO:teuthology.orchestra.run.smithi092.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 79 kB/s | 16 kB 00:00 2024-03-18T14:38:58.803 INFO:teuthology.orchestra.run.smithi092.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 567 kB/s | 90 kB 00:00 2024-03-18T14:38:58.937 INFO:teuthology.orchestra.run.smithi092.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 178 kB/s | 28 kB 00:00 2024-03-18T14:38:58.979 INFO:teuthology.orchestra.run.smithi092.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 175 kB/s | 31 kB 00:00 2024-03-18T14:38:59.097 INFO:teuthology.orchestra.run.smithi092.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-03-18T14:38:59.263 INFO:teuthology.orchestra.run.smithi092.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 208 kB/s | 59 kB 00:00 2024-03-18T14:38:59.527 INFO:teuthology.orchestra.run.smithi006.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 1.9 MB/s | 1.8 MB 00:00 2024-03-18T14:38:59.547 INFO:teuthology.orchestra.run.smithi092.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 180 kB/s | 51 kB 00:00 2024-03-18T14:38:59.789 INFO:teuthology.orchestra.run.smithi092.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 951 kB/s | 230 kB 00:00 2024-03-18T14:38:59.864 INFO:teuthology.orchestra.run.smithi092.stdout:(144/174): python3-tempora-5.0.0-2.el9.noarch.r 47 kB/s | 36 kB 00:00 2024-03-18T14:38:59.956 INFO:teuthology.orchestra.run.smithi092.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 538 kB/s | 90 kB 00:00 2024-03-18T14:38:59.990 INFO:teuthology.orchestra.run.smithi092.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-18T14:39:00.048 INFO:teuthology.orchestra.run.smithi092.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-18T14:39:00.107 INFO:teuthology.orchestra.run.smithi092.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-18T14:39:00.799 INFO:teuthology.orchestra.run.smithi092.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-18T14:39:01.891 INFO:teuthology.orchestra.run.smithi092.stdout:(150/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-03-18T14:39:01.941 INFO:teuthology.orchestra.run.smithi092.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-18T14:39:02.208 INFO:teuthology.orchestra.run.smithi092.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 8.6 MB/s | 2.3 MB 00:00 2024-03-18T14:39:02.316 INFO:teuthology.orchestra.run.smithi092.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-18T14:39:02.350 INFO:teuthology.orchestra.run.smithi092.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 42 kB 00:00 2024-03-18T14:39:02.392 INFO:teuthology.orchestra.run.smithi092.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-18T14:39:02.434 INFO:teuthology.orchestra.run.smithi092.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-18T14:39:02.476 INFO:teuthology.orchestra.run.smithi092.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 874 kB/s | 36 kB 00:00 2024-03-18T14:39:02.526 INFO:teuthology.orchestra.run.smithi092.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2024-03-18T14:39:02.577 INFO:teuthology.orchestra.run.smithi092.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-18T14:39:02.619 INFO:teuthology.orchestra.run.smithi092.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 552 kB/s | 23 kB 00:00 2024-03-18T14:39:02.660 INFO:teuthology.orchestra.run.smithi092.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 108 kB 00:00 2024-03-18T14:39:02.786 INFO:teuthology.orchestra.run.smithi092.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 155 kB 00:00 2024-03-18T14:39:02.828 INFO:teuthology.orchestra.run.smithi092.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-18T14:39:02.887 INFO:teuthology.orchestra.run.smithi092.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-18T14:39:02.953 INFO:teuthology.orchestra.run.smithi092.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 431 kB/s | 29 kB 00:00 2024-03-18T14:39:03.163 INFO:teuthology.orchestra.run.smithi092.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 5.8 MB/s | 1.2 MB 00:00 2024-03-18T14:39:03.444 INFO:teuthology.orchestra.run.smithi006.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-03-18T14:39:03.463 INFO:teuthology.orchestra.run.smithi092.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 2.2 MB 00:00 2024-03-18T14:39:03.680 INFO:teuthology.orchestra.run.smithi092.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-18T14:39:03.747 INFO:teuthology.orchestra.run.smithi092.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-03-18T14:39:03.780 INFO:teuthology.orchestra.run.smithi092.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 932 kB/s | 31 kB 00:00 2024-03-18T14:39:03.889 INFO:teuthology.orchestra.run.smithi092.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-18T14:39:03.931 INFO:teuthology.orchestra.run.smithi092.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 74 kB 00:00 2024-03-18T14:39:03.994 INFO:teuthology.orchestra.run.smithi006.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 452 kB/s | 248 kB 00:00 2024-03-18T14:39:04.086 INFO:teuthology.orchestra.run.smithi006.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 536 kB/s | 49 kB 00:00 2024-03-18T14:39:04.178 INFO:teuthology.orchestra.run.smithi006.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 179 kB/s | 16 kB 00:00 2024-03-18T14:39:04.420 INFO:teuthology.orchestra.run.smithi006.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.1 MB/s | 264 kB 00:00 2024-03-18T14:39:04.881 INFO:teuthology.orchestra.run.smithi092.stdout:(173/174): ceph-common-19.0.0-2162.g3d78f5ba.el 864 kB/s | 21 MB 00:24 2024-03-18T14:39:05.029 INFO:teuthology.orchestra.run.smithi006.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 1.2 MB/s | 748 kB 00:00 2024-03-18T14:39:05.121 INFO:teuthology.orchestra.run.smithi006.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 199 kB/s | 18 kB 00:00 2024-03-18T14:39:05.288 INFO:teuthology.orchestra.run.smithi006.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-18T14:39:05.354 INFO:teuthology.orchestra.run.smithi006.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-18T14:39:05.388 INFO:teuthology.orchestra.run.smithi006.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-18T14:39:05.431 INFO:teuthology.orchestra.run.smithi006.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-18T14:39:05.464 INFO:teuthology.orchestra.run.smithi006.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 956 kB/s | 32 kB 00:00 2024-03-18T14:39:05.589 INFO:teuthology.orchestra.run.smithi006.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-18T14:39:05.623 INFO:teuthology.orchestra.run.smithi006.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 517 kB/s | 17 kB 00:00 2024-03-18T14:39:05.657 INFO:teuthology.orchestra.run.smithi006.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-18T14:39:05.694 INFO:teuthology.orchestra.run.smithi006.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-18T14:39:05.724 INFO:teuthology.orchestra.run.smithi006.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T14:39:05.774 INFO:teuthology.orchestra.run.smithi006.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T14:39:05.833 INFO:teuthology.orchestra.run.smithi006.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 9.7 MB/s | 578 kB 00:00 2024-03-18T14:39:05.866 INFO:teuthology.orchestra.run.smithi006.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-18T14:39:05.908 INFO:teuthology.orchestra.run.smithi006.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-18T14:39:05.950 INFO:teuthology.orchestra.run.smithi006.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-18T14:39:05.992 INFO:teuthology.orchestra.run.smithi006.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-18T14:39:06.026 INFO:teuthology.orchestra.run.smithi006.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-18T14:39:06.184 INFO:teuthology.orchestra.run.smithi006.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-18T14:39:06.260 INFO:teuthology.orchestra.run.smithi006.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-18T14:39:06.310 INFO:teuthology.orchestra.run.smithi006.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-18T14:39:06.339 INFO:teuthology.orchestra.run.smithi092.stdout:(174/174): librados2-19.0.0-2162.g3d78f5ba.el9. 558 kB/s | 3.4 MB 00:06 2024-03-18T14:39:06.344 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:39:06.344 INFO:teuthology.orchestra.run.smithi092.stdout:Total 8.3 MB/s | 224 MB 00:26 2024-03-18T14:39:06.352 INFO:teuthology.orchestra.run.smithi006.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-18T14:39:06.535 INFO:teuthology.orchestra.run.smithi006.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-18T14:39:06.594 INFO:teuthology.orchestra.run.smithi006.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-18T14:39:06.620 INFO:teuthology.orchestra.run.smithi006.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 3.0 MB/s | 78 kB 00:00 2024-03-18T14:39:06.795 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-18T14:39:06.853 INFO:teuthology.orchestra.run.smithi006.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-18T14:39:06.895 INFO:teuthology.orchestra.run.smithi006.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 590 kB/s | 24 kB 00:00 2024-03-18T14:39:06.937 INFO:teuthology.orchestra.run.smithi006.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 433 kB/s | 18 kB 00:00 2024-03-18T14:39:06.979 INFO:teuthology.orchestra.run.smithi006.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-18T14:39:07.030 INFO:teuthology.orchestra.run.smithi006.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-18T14:39:07.102 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-18T14:39:07.102 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-18T14:39:07.205 INFO:teuthology.orchestra.run.smithi006.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-18T14:39:07.247 INFO:teuthology.orchestra.run.smithi006.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-18T14:39:07.297 INFO:teuthology.orchestra.run.smithi006.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-18T14:39:07.331 INFO:teuthology.orchestra.run.smithi006.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-18T14:39:07.364 INFO:teuthology.orchestra.run.smithi006.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-18T14:39:07.406 INFO:teuthology.orchestra.run.smithi006.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-18T14:39:07.457 INFO:teuthology.orchestra.run.smithi006.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T14:39:07.507 INFO:teuthology.orchestra.run.smithi006.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T14:39:07.565 INFO:teuthology.orchestra.run.smithi006.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 813 kB/s | 47 kB 00:00 2024-03-18T14:39:07.616 INFO:teuthology.orchestra.run.smithi006.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-18T14:39:08.325 INFO:teuthology.orchestra.run.smithi006.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-03-18T14:39:08.383 INFO:teuthology.orchestra.run.smithi006.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 665 kB/s | 39 kB 00:00 2024-03-18T14:39:08.433 INFO:teuthology.orchestra.run.smithi006.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 821 kB/s | 41 kB 00:00 2024-03-18T14:39:08.484 INFO:teuthology.orchestra.run.smithi006.stdout:(80/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-18T14:39:08.517 INFO:teuthology.orchestra.run.smithi006.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2024-03-18T14:39:08.551 INFO:teuthology.orchestra.run.smithi006.stdout:(82/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-18T14:39:08.584 INFO:teuthology.orchestra.run.smithi006.stdout:(83/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T14:39:08.626 INFO:teuthology.orchestra.run.smithi006.stdout:(84/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-18T14:39:08.668 INFO:teuthology.orchestra.run.smithi006.stdout:(85/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-18T14:39:08.744 INFO:teuthology.orchestra.run.smithi006.stdout:(86/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-18T14:39:08.785 INFO:teuthology.orchestra.run.smithi006.stdout:(87/174): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-18T14:39:08.828 INFO:teuthology.orchestra.run.smithi006.stdout:(88/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-18T14:39:08.870 INFO:teuthology.orchestra.run.smithi006.stdout:(89/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-18T14:39:08.996 INFO:teuthology.orchestra.run.smithi006.stdout:(90/174): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-18T14:39:09.087 INFO:teuthology.orchestra.run.smithi006.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 8.6 MB/s | 803 kB 00:00 2024-03-18T14:39:09.133 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-18T14:39:09.133 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-18T14:39:09.137 INFO:teuthology.orchestra.run.smithi006.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-18T14:39:09.179 INFO:teuthology.orchestra.run.smithi006.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-03-18T14:39:09.346 INFO:teuthology.orchestra.run.smithi006.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 129 kB/s | 21 kB 00:00 2024-03-18T14:39:09.413 INFO:teuthology.orchestra.run.smithi006.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 383 kB/s | 25 kB 00:00 2024-03-18T14:39:09.630 INFO:teuthology.orchestra.run.smithi006.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 19 MB/s | 4.1 MB 00:00 2024-03-18T14:39:09.747 INFO:teuthology.orchestra.run.smithi006.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-18T14:39:09.856 INFO:teuthology.orchestra.run.smithi006.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-18T14:39:09.923 INFO:teuthology.orchestra.run.smithi006.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-18T14:39:09.982 INFO:teuthology.orchestra.run.smithi006.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 600 kB/s | 35 kB 00:00 2024-03-18T14:39:10.057 INFO:teuthology.orchestra.run.smithi006.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 594 kB/s | 44 kB 00:00 2024-03-18T14:39:10.216 INFO:teuthology.orchestra.run.smithi006.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 592 kB/s | 94 kB 00:00 2024-03-18T14:39:10.482 INFO:teuthology.orchestra.run.smithi006.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-18T14:39:11.008 INFO:teuthology.orchestra.run.smithi006.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.6 MB/s | 863 kB 00:00 2024-03-18T14:39:11.475 INFO:teuthology.orchestra.run.smithi006.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-18T14:39:11.583 INFO:teuthology.orchestra.run.smithi006.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.8 MB/s | 309 kB 00:00 2024-03-18T14:39:11.659 INFO:teuthology.orchestra.run.smithi006.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-18T14:39:11.951 INFO:teuthology.orchestra.run.smithi006.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-18T14:39:12.026 INFO:teuthology.orchestra.run.smithi006.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-18T14:39:12.101 INFO:teuthology.orchestra.run.smithi006.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 650 kB/s | 49 kB 00:00 2024-03-18T14:39:12.176 INFO:teuthology.orchestra.run.smithi006.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 899 kB/s | 67 kB 00:00 2024-03-18T14:39:12.377 INFO:teuthology.orchestra.run.smithi006.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 755 kB/s | 151 kB 00:00 2024-03-18T14:39:12.461 INFO:teuthology.orchestra.run.smithi006.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-18T14:39:12.553 INFO:teuthology.orchestra.run.smithi006.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-03-18T14:39:12.628 INFO:teuthology.orchestra.run.smithi006.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 579 kB/s | 43 kB 00:00 2024-03-18T14:39:12.703 INFO:teuthology.orchestra.run.smithi006.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 430 kB/s | 32 kB 00:00 2024-03-18T14:39:12.779 INFO:teuthology.orchestra.run.smithi006.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-03-18T14:39:12.854 INFO:teuthology.orchestra.run.smithi006.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-18T14:39:12.929 INFO:teuthology.orchestra.run.smithi006.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-18T14:39:13.004 INFO:teuthology.orchestra.run.smithi006.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-18T14:39:13.080 INFO:teuthology.orchestra.run.smithi006.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 470 kB/s | 35 kB 00:00 2024-03-18T14:39:13.155 INFO:teuthology.orchestra.run.smithi006.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-18T14:39:13.267 INFO:teuthology.orchestra.run.smithi006.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-03-18T14:39:13.354 INFO:teuthology.orchestra.run.smithi006.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.6 MB/s | 144 kB 00:00 2024-03-18T14:39:13.431 INFO:teuthology.orchestra.run.smithi006.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 140 kB/s | 11 kB 00:00 2024-03-18T14:39:13.506 INFO:teuthology.orchestra.run.smithi006.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2024-03-18T14:39:13.573 INFO:teuthology.orchestra.run.smithi006.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-03-18T14:39:13.640 INFO:teuthology.orchestra.run.smithi006.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 291 kB/s | 19 kB 00:00 2024-03-18T14:39:13.707 INFO:teuthology.orchestra.run.smithi006.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-03-18T14:39:13.792 INFO:teuthology.orchestra.run.smithi006.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-18T14:39:13.858 INFO:teuthology.orchestra.run.smithi006.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 694 kB/s | 46 kB 00:00 2024-03-18T14:39:13.933 INFO:teuthology.orchestra.run.smithi006.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-18T14:39:14.000 INFO:teuthology.orchestra.run.smithi006.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 867 kB/s | 58 kB 00:00 2024-03-18T14:39:14.075 INFO:teuthology.orchestra.run.smithi006.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-03-18T14:39:14.142 INFO:teuthology.orchestra.run.smithi006.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-03-18T14:39:14.218 INFO:teuthology.orchestra.run.smithi006.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-18T14:39:14.284 INFO:teuthology.orchestra.run.smithi006.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 424 kB/s | 28 kB 00:00 2024-03-18T14:39:14.351 INFO:teuthology.orchestra.run.smithi006.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 461 kB/s | 31 kB 00:00 2024-03-18T14:39:14.428 INFO:teuthology.orchestra.run.smithi006.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-03-18T14:39:14.502 INFO:teuthology.orchestra.run.smithi006.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 789 kB/s | 59 kB 00:00 2024-03-18T14:39:14.578 INFO:teuthology.orchestra.run.smithi006.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-03-18T14:39:14.653 INFO:teuthology.orchestra.run.smithi006.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 681 kB/s | 51 kB 00:00 2024-03-18T14:39:14.728 INFO:teuthology.orchestra.run.smithi006.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-18T14:39:14.804 INFO:teuthology.orchestra.run.smithi006.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-18T14:39:14.879 INFO:teuthology.orchestra.run.smithi006.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-18T14:39:14.954 INFO:teuthology.orchestra.run.smithi006.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-03-18T14:39:15.030 INFO:teuthology.orchestra.run.smithi006.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-18T14:39:15.130 INFO:teuthology.orchestra.run.smithi006.stdout:(148/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-18T14:39:16.042 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-18T14:39:16.144 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T14:39:16.180 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T14:39:16.226 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:39:16.235 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:39:16.279 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T14:39:16.330 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T14:39:16.363 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T14:39:16.397 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T14:39:16.447 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T14:39:16.776 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T14:39:16.832 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:39:16.936 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:39:16.966 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:39:17.041 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:39:17.074 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:39:17.122 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T14:39:17.303 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T14:39:17.351 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T14:39:17.380 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:39:17.486 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:39:17.514 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:39:17.563 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:39:17.593 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T14:39:17.624 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:39:17.640 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:39:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:39:17.860 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:39:17.899 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T14:39:18.070 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T14:39:18.177 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:39:18.236 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:39:18.356 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:39:18.405 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:39:18.613 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:39:18.721 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T14:39:18.813 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T14:39:18.860 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T14:39:18.887 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T14:39:18.931 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T14:39:19.004 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T14:39:19.058 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T14:39:19.096 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T14:39:19.161 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T14:39:19.204 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T14:39:19.256 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T14:39:19.289 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T14:39:19.316 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T14:39:19.384 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T14:39:19.421 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T14:39:19.471 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T14:39:19.510 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T14:39:19.578 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T14:39:19.624 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T14:39:19.706 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T14:39:19.847 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T14:39:19.911 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T14:39:19.963 INFO:teuthology.orchestra.run.smithi006.stdout:(149/174): librados2-19.0.0-2162.g3d78f5ba.el9. 719 kB/s | 3.4 MB 00:04 2024-03-18T14:39:20.427 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T14:39:20.505 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T14:39:20.598 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T14:39:20.651 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:39:20.659 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:39:20.692 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T14:39:20.740 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T14:39:20.778 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T14:39:20.809 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T14:39:21.067 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T14:39:21.250 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T14:39:21.337 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T14:39:21.380 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T14:39:21.540 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T14:39:21.584 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T14:39:21.638 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T14:39:21.710 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T14:39:21.743 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T14:39:21.779 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T14:39:21.826 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T14:39:21.870 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T14:39:21.921 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T14:39:21.963 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T14:39:22.096 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T14:39:22.144 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T14:39:22.175 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T14:39:22.216 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T14:39:22.309 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T14:39:22.358 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T14:39:22.397 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T14:39:22.423 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T14:39:22.464 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T14:39:22.553 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T14:39:22.680 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T14:39:22.780 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T14:39:22.831 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T14:39:22.879 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T14:39:22.935 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T14:39:23.116 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T14:39:23.160 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T14:39:23.314 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T14:39:23.358 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T14:39:23.389 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T14:39:23.420 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T14:39:23.448 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T14:39:23.513 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T14:39:23.556 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T14:39:23.587 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T14:39:23.617 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T14:39:23.692 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T14:39:23.736 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T14:39:23.769 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T14:39:23.810 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T14:39:23.845 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T14:39:23.934 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T14:39:24.023 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T14:39:24.068 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T14:39:24.108 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T14:39:24.154 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T14:39:24.253 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T14:39:24.309 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T14:39:25.022 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T14:39:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T14:39:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T14:39:25.162 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T14:39:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T14:39:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T14:39:25.449 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T14:39:25.496 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T14:39:25.598 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T14:39:26.235 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T14:39:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T14:39:27.330 INFO:teuthology.orchestra.run.smithi006.stdout:(150/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 437 kB/s | 3.1 MB 00:07 2024-03-18T14:39:27.371 INFO:teuthology.orchestra.run.smithi006.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-18T14:39:27.570 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T14:39:27.605 INFO:teuthology.orchestra.run.smithi006.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 9.8 MB/s | 2.3 MB 00:00 2024-03-18T14:39:27.646 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T14:39:27.677 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T14:39:27.680 INFO:teuthology.orchestra.run.smithi006.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-18T14:39:27.708 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T14:39:27.722 INFO:teuthology.orchestra.run.smithi006.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-18T14:39:27.735 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T14:39:27.772 INFO:teuthology.orchestra.run.smithi006.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.9 MB/s | 248 kB 00:00 2024-03-18T14:39:27.815 INFO:teuthology.orchestra.run.smithi006.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-18T14:39:27.856 INFO:teuthology.orchestra.run.smithi006.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 879 kB/s | 36 kB 00:00 2024-03-18T14:39:28.032 INFO:teuthology.orchestra.run.smithi006.stdout:(158/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 1.3 MB/s | 48 MB 00:37 2024-03-18T14:39:28.046 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T14:39:28.057 INFO:teuthology.orchestra.run.smithi006.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 959 kB/s | 192 kB 00:00 2024-03-18T14:39:28.074 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:39:28.091 INFO:teuthology.orchestra.run.smithi006.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 693 kB/s | 23 kB 00:00 2024-03-18T14:39:28.132 INFO:teuthology.orchestra.run.smithi006.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 108 kB 00:00 2024-03-18T14:39:28.147 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:39:28.174 INFO:teuthology.orchestra.run.smithi006.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-03-18T14:39:28.178 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T14:39:28.200 INFO:teuthology.orchestra.run.smithi006.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.5 MB/s | 256 kB 00:00 2024-03-18T14:39:28.223 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T14:39:28.225 INFO:teuthology.orchestra.run.smithi006.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-18T14:39:28.242 INFO:teuthology.orchestra.run.smithi006.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-18T14:39:28.259 INFO:teuthology.orchestra.run.smithi006.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 852 kB/s | 29 kB 00:00 2024-03-18T14:39:28.351 INFO:teuthology.orchestra.run.smithi006.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 23 MB/s | 2.2 MB 00:00 2024-03-18T14:39:28.393 INFO:teuthology.orchestra.run.smithi006.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.0 MB/s | 1.2 MB 00:00 2024-03-18T14:39:28.474 INFO:teuthology.orchestra.run.smithi006.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 19 MB/s | 2.3 MB 00:00 2024-03-18T14:39:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-18T14:39:28.519 INFO:teuthology.orchestra.run.smithi006.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 696 kB/s | 31 kB 00:00 2024-03-18T14:39:28.586 INFO:teuthology.orchestra.run.smithi006.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-18T14:39:28.689 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T14:39:28.718 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:39:28.789 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:39:28.820 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T14:39:28.870 INFO:teuthology.orchestra.run.smithi006.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 212 kB/s | 74 kB 00:00 2024-03-18T14:39:31.045 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:39:31.094 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:39:31.628 INFO:teuthology.orchestra.run.smithi006.stdout:(174/174): ceph-common-19.0.0-2162.g3d78f5ba.el 448 kB/s | 21 MB 00:47 2024-03-18T14:39:31.633 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:39:31.633 INFO:teuthology.orchestra.run.smithi006.stdout:Total 4.7 MB/s | 224 MB 00:47 2024-03-18T14:39:31.649 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:39:31.853 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:39:31.942 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:39:31.942 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T14:39:31.942 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T14:39:31.942 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:31.979 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:39:32.115 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-18T14:39:32.447 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-18T14:39:32.447 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-18T14:39:34.579 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-18T14:39:34.579 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-18T14:39:41.794 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-18T14:39:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T14:39:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T14:39:41.972 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:39:41.981 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T14:39:42.029 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T14:39:42.088 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T14:39:42.133 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T14:39:42.158 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T14:39:42.208 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T14:39:42.554 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T14:39:42.600 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:39:42.732 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:39:42.876 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:39:42.978 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:39:43.013 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:39:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T14:39:43.232 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T14:39:43.287 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T14:39:43.314 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:39:43.423 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:39:43.449 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:39:43.506 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T14:39:43.545 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T14:39:43.567 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:39:43.583 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T14:39:43.712 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2024-03-18T14:39:43.713 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:43.776 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:39:43.818 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T14:39:43.849 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T14:39:43.873 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:39:43.888 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:39:43.888 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:43.889 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T14:39:43.889 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:44.037 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T14:39:44.174 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:39:44.236 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T14:39:44.341 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:39:44.404 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:39:44.635 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T14:39:44.755 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T14:39:44.892 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T14:39:44.931 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T14:39:44.962 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T14:39:45.005 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T14:39:45.082 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T14:39:45.134 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T14:39:45.171 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T14:39:45.238 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T14:39:45.271 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T14:39:45.314 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T14:39:45.347 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T14:39:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T14:39:45.376 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:39:45.401 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:39:45.401 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:45.401 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T14:39:45.401 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:39:45.401 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:39:45.402 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:45.445 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T14:39:45.485 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T14:39:45.527 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T14:39:45.551 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T14:39:45.623 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T14:39:45.627 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:39:45.649 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:39:45.649 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:45.649 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T14:39:45.650 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:39:45.650 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:39:45.650 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:45.666 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T14:39:45.743 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T14:39:45.888 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T14:39:45.945 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T14:39:46.086 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:39:46.108 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:39:46.108 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:46.108 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T14:39:46.108 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:39:46.108 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:39:46.108 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:46.204 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T14:39:46.498 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T14:39:46.582 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T14:39:46.630 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:39:46.657 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T14:39:46.661 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:39:46.693 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T14:39:46.708 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:39:46.717 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T14:39:46.752 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T14:39:46.755 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T14:39:46.776 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:39:46.798 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T14:39:46.827 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T14:39:46.857 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T14:39:47.010 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:39:47.043 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:39:47.130 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T14:39:47.321 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T14:39:47.428 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T14:39:47.472 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T14:39:47.617 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T14:39:47.658 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T14:39:47.715 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T14:39:47.788 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T14:39:47.833 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T14:39:47.871 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T14:39:47.917 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T14:39:47.953 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T14:39:47.996 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T14:39:48.006 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:39:48.038 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T14:39:48.043 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:39:48.144 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:39:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T14:39:48.226 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T14:39:48.274 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T14:39:48.306 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T14:39:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T14:39:48.337 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:39:48.360 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:39:48.360 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:48.360 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T14:39:48.360 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:39:48.360 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:39:48.360 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:48.404 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:39:48.412 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T14:39:48.415 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:39:48.448 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T14:39:48.476 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T14:39:48.489 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T14:39:48.514 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T14:39:48.538 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T14:39:48.619 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T14:39:48.743 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T14:39:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T14:39:48.905 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T14:39:48.963 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T14:39:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T14:39:49.219 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T14:39:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T14:39:49.348 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T14:39:49.381 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:39:49.404 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:39:49.404 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:49.404 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T14:39:49.404 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:39:49.404 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:39:49.404 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:49.413 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T14:39:49.448 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:39:49.449 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T14:39:49.470 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:39:49.470 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:49.470 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T14:39:49.470 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:49.480 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T14:39:49.511 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T14:39:49.524 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:39:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T14:39:49.546 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:39:49.546 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:49.547 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T14:39:49.547 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:39:49.547 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:39:49.547 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:49.566 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T14:39:49.621 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T14:39:49.653 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T14:39:49.683 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T14:39:49.763 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T14:39:49.802 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T14:39:49.834 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T14:39:49.858 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:39:49.877 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T14:39:49.881 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:39:49.881 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:39:49.881 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T14:39:49.881 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:39:49.881 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:39:49.881 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:39:49.911 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T14:39:49.994 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T14:39:50.084 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T14:39:50.125 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T14:39:50.158 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T14:39:50.213 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T14:39:50.322 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T14:39:50.375 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T14:39:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T14:39:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T14:39:51.230 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T14:39:51.297 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T14:39:51.395 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T14:39:51.431 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T14:39:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T14:39:51.636 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T14:39:51.716 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T14:39:52.390 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T14:39:52.512 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T14:39:53.793 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T14:39:53.887 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T14:39:53.918 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T14:39:53.949 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T14:39:53.976 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T14:39:54.305 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T14:39:54.331 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:39:54.337 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:39:54.370 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T14:39:54.408 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T14:39:54.422 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T14:39:54.436 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T14:39:54.452 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T14:39:54.489 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T14:39:54.617 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T14:39:54.727 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T14:39:54.767 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T14:39:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T14:39:54.894 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T14:39:54.935 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T14:39:54.968 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T14:39:54.977 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T14:39:55.000 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T14:39:55.000 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:39:55.030 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T14:39:55.071 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T14:39:55.093 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T14:39:55.102 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T14:39:55.171 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T14:39:55.203 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T14:39:55.845 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:39:55.884 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:39:55.904 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:39:56.714 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T14:39:56.796 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T14:39:56.837 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T14:39:56.934 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T14:39:56.974 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T14:39:56.999 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T14:39:57.024 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T14:39:57.051 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T14:39:57.052 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:39:57.132 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:39:57.162 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T14:39:57.162 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:39:57.230 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:39:57.265 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T14:39:57.306 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T14:39:57.332 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T14:39:57.357 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T14:39:57.403 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T14:39:57.433 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:39:57.446 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T14:39:57.465 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:39:57.494 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T14:39:57.520 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T14:39:57.545 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T14:39:57.570 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:39:57.596 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T14:39:57.602 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:39:57.602 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:39:57.782 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:39:57.782 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:39:57.865 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:39:57.889 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:39:57.923 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T14:39:57.957 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T14:39:57.982 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T14:39:58.038 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T14:39:58.048 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:39:58.076 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:39:58.175 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T14:39:58.175 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T14:39:58.175 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T14:39:58.175 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:39:58.220 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:40:02.800 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:40:02.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T14:40:02.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T14:40:02.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T14:40:02.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:02.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T14:40:02.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T14:40:02.817 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T14:40:02.817 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T14:40:02.818 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T14:40:02.819 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T14:40:02.820 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T14:40:02.821 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T14:40:02.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T14:40:02.823 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T14:40:02.824 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T14:40:02.825 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T14:40:02.826 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T14:40:02.827 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T14:40:02.828 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T14:40:02.829 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T14:40:05.713 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.714 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.715 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T14:40:05.717 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-16.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.718 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T14:40:05.719 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.720 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T14:40:05.721 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T14:40:05.722 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T14:40:05.723 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T14:40:05.725 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.726 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T14:40:05.727 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:05.728 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-18T14:40:06.222 DEBUG:teuthology.parallel:result is None 2024-03-18T14:40:10.146 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-03-18T14:40:10.147 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:10.316 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:40:10.331 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T14:40:10.331 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:10.331 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T14:40:10.331 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:11.869 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:40:11.894 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T14:40:11.894 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:11.895 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T14:40:11.895 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:40:11.895 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T14:40:11.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:12.128 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:40:12.150 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T14:40:12.151 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:12.151 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T14:40:12.151 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:40:12.151 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T14:40:12.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:12.612 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:40:12.634 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T14:40:12.634 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:12.634 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T14:40:12.634 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:40:12.634 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T14:40:12.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:12.722 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T14:40:13.114 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:40:13.146 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T14:40:13.178 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T14:40:13.240 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T14:40:13.268 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:40:13.513 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T14:40:13.544 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:40:14.559 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T14:40:14.603 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:40:14.710 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T14:40:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T14:40:14.905 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:40:14.927 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T14:40:14.927 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:14.927 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T14:40:14.928 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:40:14.928 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T14:40:14.928 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:14.972 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:40:14.983 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T14:40:15.038 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T14:40:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T14:40:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:40:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T14:40:16.014 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:16.014 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T14:40:16.014 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:40:16.015 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T14:40:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:16.058 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:40:16.080 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T14:40:16.080 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:16.080 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T14:40:16.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:16.125 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:40:16.147 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:40:16.148 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:16.148 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T14:40:16.148 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:40:16.148 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T14:40:16.148 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:16.468 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:40:16.490 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T14:40:16.490 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T14:40:16.490 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T14:40:16.490 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:40:16.490 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T14:40:16.490 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:21.189 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:40:21.240 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T14:40:21.269 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T14:40:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T14:40:21.495 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T14:40:21.643 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T14:40:21.687 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T14:40:21.738 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T14:40:21.786 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T14:40:21.821 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T14:40:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T14:40:21.885 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T14:40:21.915 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T14:40:21.980 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T14:40:22.053 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T14:40:22.089 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T14:40:22.663 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:40:22.704 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:40:22.723 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T14:40:23.600 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T14:40:23.681 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T14:40:23.723 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T14:40:23.806 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T14:40:23.835 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T14:40:23.860 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T14:40:23.885 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T14:40:23.911 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T14:40:23.912 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:40:23.983 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T14:40:24.013 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T14:40:24.013 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:40:24.089 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T14:40:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T14:40:24.148 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T14:40:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T14:40:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T14:40:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T14:40:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T14:40:24.312 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T14:40:24.338 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T14:40:24.363 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T14:40:24.388 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:40:24.413 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T14:40:24.420 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:40:24.420 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:40:24.608 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T14:40:24.608 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:40:24.690 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T14:40:24.715 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:40:24.740 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T14:40:24.774 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T14:40:24.800 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T14:40:24.868 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T14:40:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T14:40:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T14:40:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T14:40:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T14:40:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T14:40:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T14:40:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T14:40:29.679 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T14:40:29.700 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T14:40:29.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T14:40:29.702 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T14:40:29.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T14:40:29.704 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T14:40:29.705 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T14:40:29.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T14:40:29.707 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T14:40:29.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T14:40:29.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T14:40:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T14:40:29.711 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T14:40:29.712 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T14:40:29.713 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T14:40:29.713 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T14:40:29.713 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T14:40:29.713 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T14:40:29.713 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T14:40:29.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T14:40:29.715 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T14:40:29.715 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T14:40:29.715 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T14:40:29.715 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T14:40:29.715 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T14:40:29.715 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T14:40:29.715 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T14:40:29.716 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T14:40:29.717 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T14:40:29.717 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T14:40:29.717 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T14:40:29.717 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T14:40:29.717 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T14:40:29.717 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T14:40:29.717 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T14:40:32.706 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T14:40:32.706 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:32.706 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-18T14:40:32.706 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:32.706 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:32.706 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:32.706 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.707 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T14:40:32.708 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.710 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T14:40:32.711 INFO:teuthology.orchestra.run.smithi006.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.712 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T14:40:32.713 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T14:40:32.714 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.715 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T14:40:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T14:40:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T14:40:32.718 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T14:40:32.720 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T14:40:32.721 INFO:teuthology.orchestra.run.smithi006.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T14:40:32.722 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T14:40:32.722 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T14:40:32.722 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T14:40:32.723 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:32.723 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-18T14:40:33.201 DEBUG:teuthology.parallel:result is None 2024-03-18T14:40:33.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:40:33.345 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T14:40:33.368 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:40:33.369 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:40:33.369 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T14:40:33.371 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:40:33.519 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T14:40:33.545 INFO:teuthology.orchestra.run.smithi092.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:40:33.546 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T14:40:33.546 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T14:40:33.550 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T14:40:33.550 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:33.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T14:40:33.582 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:33.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T14:40:33.618 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T14:40:33.618 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:33.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T14:40:33.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T14:40:33.720 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:33.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T14:40:33.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T14:40:33.814 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T14:40:33.815 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:33.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T14:40:33.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T14:40:33.910 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:33.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T14:40:33.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T14:40:34.005 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T14:40:34.005 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:34.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T14:40:34.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T14:40:34.100 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:34.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T14:40:34.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T14:40:34.194 INFO:teuthology.run_tasks:Running task ceph... 2024-03-18T14:40:34.282 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-18T14:40:34.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-03-18T14:40:34.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2024-03-18T14:40:34.311 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-18T14:40:34.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-18T14:40:34.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-18T14:40:34.379 INFO:tasks.ceph:Creating extra log directories... 2024-03-18T14:40:34.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-18T14:40:34.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-18T14:40:34.455 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-18T14:40:34.455 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-18T14:40:34.455 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-18T14:40:34.456 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:34.456 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-18T14:40:34.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-18T14:40:34.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-18T14:40:34.649 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:34.649 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-18T14:40:34.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-18T14:40:34.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-18T14:40:34.813 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-18T14:40:34.814 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-18T14:40:34.814 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831', 'branch': 'wip-batrick-testing-20240314.183243', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7608831', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240314.183243'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240314.183243', 'sha1': '4e19f704afd9957ea657ab858f9ecf76401fef6f'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4770, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'sleep_before_teardown': 0, 'subset': '498/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240314.183243', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': '4e19f704afd9957ea657ab858f9ecf76401fef6f', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQtYpzYGP9RtuDnRfRAjMIFbIwY8l0IVJ419joOtQa91wnFCza3eQSAAxQj5WQX7MEHuvT3LFPqwb5o4+jEM8M=', 'smithi092.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1btTUr4MtuN4TIKkNYtJ8gv15N6zwZK5VqgoONnMI+bye+UznISZRSsJ0R/IXGCamLgfoAwgbJ30P0fFhjZ3g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-18_14:19:46', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-18T14:40:34.814 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-18T14:40:34.817 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-18T14:40:34.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-03-18T14:40:34.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2024-03-18T14:40:34.930 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:34.930 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T14:40:34.952 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T14:40:34.953 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-03-18T14:40:35.007 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T14:40:35.007 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:35.008 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-18T14:40:35.008 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:35.008 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:35.008 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-18 14:40:25.216110202 +0000 2024-03-18T14:40:35.008 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-18 14:37:45.871022364 +0000 2024-03-18T14:40:35.008 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-18 14:37:45.871022364 +0000 2024-03-18T14:40:35.008 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-18 14:37:45.871022364 +0000 2024-03-18T14:40:35.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T14:40:35.073 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-18T14:40:35.073 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-18T14:40:35.073 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000188871 s, 2.7 MB/s 2024-03-18T14:40:35.074 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T14:40:35.130 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-18 14:40:25.216110202 +0000 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-18 14:37:46.230018331 +0000 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-18 14:37:46.230018331 +0000 2024-03-18T14:40:35.185 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-18 14:37:46.230018331 +0000 2024-03-18T14:40:35.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T14:40:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-18T14:40:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-18T14:40:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000159278 s, 3.2 MB/s 2024-03-18T14:40:35.250 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T14:40:35.306 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-18 14:40:25.217110191 +0000 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-18 14:37:46.571014500 +0000 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-18 14:37:46.571014500 +0000 2024-03-18T14:40:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-18 14:37:46.571014500 +0000 2024-03-18T14:40:35.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T14:40:35.424 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-18T14:40:35.424 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-18T14:40:35.424 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000202244 s, 2.5 MB/s 2024-03-18T14:40:35.425 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T14:40:35.480 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-18 14:40:25.217110191 +0000 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-18 14:37:46.947010276 +0000 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-18 14:37:46.947010276 +0000 2024-03-18T14:40:35.535 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-18 14:37:46.947010276 +0000 2024-03-18T14:40:35.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T14:40:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-18T14:40:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-18T14:40:35.601 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000190624 s, 2.7 MB/s 2024-03-18T14:40:35.602 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T14:40:35.657 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:35.657 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:35.657 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T14:40:35.674 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T14:40:35.674 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-03-18T14:40:35.730 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-18 14:39:58.381712191 +0000 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-18 14:37:44.921331831 +0000 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-18 14:37:44.921331831 +0000 2024-03-18T14:40:35.731 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-18 14:37:44.921331831 +0000 2024-03-18T14:40:35.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T14:40:35.797 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-18T14:40:35.797 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-18T14:40:35.797 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000212886 s, 2.4 MB/s 2024-03-18T14:40:35.798 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T14:40:35.855 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-03-18T14:40:35.910 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T14:40:35.910 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:35.910 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-18T14:40:35.911 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:35.911 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:35.911 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-18 14:39:58.382712163 +0000 2024-03-18T14:40:35.911 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-18 14:37:45.288321248 +0000 2024-03-18T14:40:35.911 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-18 14:37:45.288321248 +0000 2024-03-18T14:40:35.911 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-18 14:37:45.288321248 +0000 2024-03-18T14:40:35.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T14:40:35.976 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-18T14:40:35.976 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-18T14:40:35.976 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000189257 s, 2.7 MB/s 2024-03-18T14:40:35.977 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T14:40:36.033 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-18 14:39:58.382712163 +0000 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-18 14:37:45.627311473 +0000 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-18 14:37:45.627311473 +0000 2024-03-18T14:40:36.087 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-18 14:37:45.627311473 +0000 2024-03-18T14:40:36.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T14:40:36.153 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-18T14:40:36.153 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-18T14:40:36.153 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000262819 s, 1.9 MB/s 2024-03-18T14:40:36.154 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T14:40:36.210 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-03-18T14:40:36.265 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T14:40:36.265 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T14:40:36.265 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-18T14:40:36.265 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T14:40:36.265 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T14:40:36.265 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-18 14:39:58.382712163 +0000 2024-03-18T14:40:36.265 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-18 14:37:45.979301323 +0000 2024-03-18T14:40:36.266 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-18 14:37:45.979301323 +0000 2024-03-18T14:40:36.266 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-18 14:37:45.979301323 +0000 2024-03-18T14:40:36.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T14:40:36.330 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-18T14:40:36.331 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-18T14:40:36.331 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000242262 s, 2.1 MB/s 2024-03-18T14:40:36.332 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T14:40:36.388 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:36.400 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-18T14:40:36.400 INFO:tasks.ceph:Generating config... 2024-03-18T14:40:36.401 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-18T14:40:36.401 INFO:tasks.ceph:[client] debug client = 20 2024-03-18T14:40:36.401 INFO:tasks.ceph:[client] debug ms = 1 2024-03-18T14:40:36.402 INFO:tasks.ceph:[client] pid file = 2024-03-18T14:40:36.402 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-18T14:40:36.402 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-18T14:40:36.402 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-18T14:40:36.403 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-18T14:40:36.403 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-18T14:40:36.403 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-18T14:40:36.403 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-18T14:40:36.403 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-18T14:40:36.403 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-18T14:40:36.404 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-18T14:40:36.404 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-18T14:40:36.404 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-18T14:40:36.404 INFO:tasks.ceph:Setting up mon.a... 2024-03-18T14:40:36.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-18T14:40:36.456 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-03-18T14:40:36.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-18T14:40:36.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T14:40:36.531 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.b', '172.21.15.92'), ('mon.c', '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]')] 2024-03-18T14:40:36.532 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-18T14:40:36.533 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:36.533 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-18T14:40:36.587 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --add b 172.21.15.92 --addv c '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:36.672 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid a75a8f99-9e80-46f7-9da4-368af92e8063 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:fsid a75a8f99-9e80-46f7-9da4-368af92e8063 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-03-18T14:40:36.671372+0000 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-03-18T14:40:36.671372+0000 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-03-18T14:40:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-03-18T14:40:36.674 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-18T14:40:36.675 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-18T14:40:36.689 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a75a8f99-9e80-46f7-9da4-368af92e8063... 2024-03-18T14:40:36.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T14:40:36.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T14:40:36.779 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-18T14:40:36.779 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-18T14:40:36.779 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.780 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-18T14:40:36.781 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-18T14:40:36.782 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = a75a8f99-9e80-46f7-9da4-368af92e8063 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-18T14:40:36.783 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.784 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-18T14:40:36.785 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-18T14:40:36.786 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T14:40:36.787 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-18T14:40:36.788 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-18T14:40:36.789 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-18T14:40:36.790 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2024-03-18T14:40:36.791 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T14:40:36.792 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T14:40:36.793 INFO:teuthology.orchestra.run.smithi092.stdout: mon_allow_pool_size_one = true 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.6,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = a75a8f99-9e80-46f7-9da4-368af92e8063 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.794 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock profile = high_recovery_ops 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug trim objects = true 2024-03-18T14:40:36.795 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2024-03-18T14:40:36.796 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore block size = 96636764160 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore fsck on mount = True 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluefs = 20 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluestore = 20 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: debug rocksdb = 10 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd full ratio = 0.9 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd nearfull ratio = 0.8 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: osd failsafe full ratio = 0.95 2024-03-18T14:40:36.797 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = bluestore 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.798 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T14:40:36.799 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: # 1m isn't quite enough 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: mon_down_mkfs_grace = 2m 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_filestore_osds = false 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: mon op complaint time = 120 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2024-03-18T14:40:36.800 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout: client mount timeout = 600 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = "" 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2024-03-18T14:40:36.801 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout:[mds] 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds = 20 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds balancer = 20 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug frag = True 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug scatterstat = True 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: mds op complaint time = 180 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: mds verify scatter = True 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2024-03-18T14:40:36.802 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2024-03-18T14:40:36.804 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-18T14:40:36.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-18T14:40:36.898 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-18T14:40:36.899 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:36.899 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-18T14:40:36.913 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:36.913 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-18T14:40:36.967 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:40:36.968 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:36.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-18T14:40:36.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T14:40:37.046 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:40:37.047 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:37.101 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:40:37.101 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:37.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-18T14:40:37.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T14:40:37.141 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:40:37.141 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:37.196 INFO:tasks.ceph:Setting up mon nodes... 2024-03-18T14:40:37.196 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-18T14:40:37.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-18T14:40:37.251 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-18T14:40:37.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-18T14:40:37.330 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-18T14:40:37.332 INFO:tasks.ceph:Setting up mds nodes... 2024-03-18T14:40:37.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-18T14:40:37.387 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-18T14:40:37.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-18T14:40:37.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-18T14:40:37.509 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-18T14:40:37.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-18T14:40:37.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-18T14:40:37.594 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-18T14:40:37.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-18T14:40:37.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-18T14:40:37.717 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-18T14:40:37.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-18T14:40:37.745 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-18T14:40:37.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T14:40:37.787 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-18T14:40:37.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-18T14:40:37.883 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-18T14:40:37.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-18T14:40:37.940 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-18T14:40:37.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-18T14:40:38.037 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-18T14:40:38.052 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-18T14:40:38.053 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-18T14:40:38.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-18T14:40:38.079 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:38.080 INFO:tasks.ceph:role: osd.0 2024-03-18T14:40:38.080 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:40:38.080 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-18T14:40:38.153 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:38.154 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:38.666 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-18T14:40:38.667 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-18T14:40:38.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-18T14:40:38.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-18T14:40:38.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-18T14:40:38.877 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:38.878 INFO:tasks.ceph:role: osd.1 2024-03-18T14:40:38.878 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:40:38.878 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:38.950 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:39.414 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-18T14:40:39.417 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-18T14:40:39.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-18T14:40:39.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-18T14:40:39.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-18T14:40:39.608 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:39.608 INFO:tasks.ceph:role: osd.2 2024-03-18T14:40:39.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:40:39.608 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:39.683 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:39.684 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:40.176 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-18T14:40:40.178 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-18T14:40:40.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-18T14:40:40.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-18T14:40:40.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-18T14:40:40.427 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:40.427 INFO:tasks.ceph:role: osd.3 2024-03-18T14:40:40.427 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:40:40.427 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-18T14:40:40.503 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:40.503 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:40.503 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:40.504 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:40.504 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:40.504 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:40.504 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:40.504 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:40.504 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:40.504 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:41.010 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-18T14:40:41.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-18T14:40:41.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-18T14:40:41.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-18T14:40:41.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:41.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:41.252+0000 7fd82ca24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-18T14:40:41.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:41.252+0000 7fd82ca24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-18T14:40:41.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:41.252+0000 7fd82ca24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T14:40:41.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:41.252+0000 7fd82ca24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T14:40:41.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:41.252+0000 7fd82ca24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T14:40:41.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:41.253+0000 7fd82ca24640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-18T14:40:44.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-18T14:40:44.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:44.229+0000 7fc307410640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-18T14:40:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:44.229+0000 7fc307410640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-18T14:40:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:44.229+0000 7fc307410640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T14:40:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:44.229+0000 7fc307410640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T14:40:44.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:44.229+0000 7fc307410640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T14:40:44.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:44.230+0000 7fc307410640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-18T14:40:47.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-18T14:40:47.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:47.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:47.199+0000 7f5d5f326640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-18T14:40:47.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:47.199+0000 7f5d5f326640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-18T14:40:47.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:47.199+0000 7f5d5f326640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T14:40:47.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:47.200+0000 7f5d5f326640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T14:40:47.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:47.200+0000 7f5d5f326640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T14:40:47.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:47.200+0000 7f5d5f326640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-18T14:40:50.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-18T14:40:50.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:50.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:50.166+0000 7efdf9a0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-18T14:40:50.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:50.167+0000 7efdf9a0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-18T14:40:50.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:50.167+0000 7efdf9a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T14:40:50.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:50.167+0000 7efdf9a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T14:40:50.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:50.167+0000 7efdf9a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T14:40:50.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:40:50.167+0000 7efdf9a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-18T14:40:53.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-18T14:40:53.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-18T14:40:53.082 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:53.082 INFO:tasks.ceph:role: osd.4 2024-03-18T14:40:53.082 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:40:53.082 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-18T14:40:53.157 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:53.158 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:53.662 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-18T14:40:53.664 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-18T14:40:53.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-18T14:40:53.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-18T14:40:53.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-18T14:40:53.914 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:53.914 INFO:tasks.ceph:role: osd.5 2024-03-18T14:40:53.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:40:53.914 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:53.993 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:54.445 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-18T14:40:54.447 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-18T14:40:54.447 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-18T14:40:54.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-18T14:40:54.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-18T14:40:54.632 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:54.632 INFO:tasks.ceph:role: osd.6 2024-03-18T14:40:54.632 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:40:54.632 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:54.707 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:55.197 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-18T14:40:55.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-18T14:40:55.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-18T14:40:55.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-18T14:40:55.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-18T14:40:55.430 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T14:40:55.430 INFO:tasks.ceph:role: osd.7 2024-03-18T14:40:55.430 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:40:55.430 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-18T14:40:55.505 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T14:40:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T14:40:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T14:40:55.505 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T14:40:55.506 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T14:40:55.506 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-18T14:40:55.506 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T14:40:55.506 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T14:40:55.506 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T14:40:55.506 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T14:40:56.049 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-18T14:40:56.051 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-18T14:40:56.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-18T14:40:56.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-18T14:40:56.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:56.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:56.311+0000 7fe77e811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-18T14:40:56.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:56.312+0000 7fe77e811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-18T14:40:56.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:56.312+0000 7fe77e811640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T14:40:56.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:56.312+0000 7fe77e811640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T14:40:56.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:56.312+0000 7fe77e811640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T14:40:56.314 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:56.312+0000 7fe77e811640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-18T14:40:59.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-18T14:40:59.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:40:59.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:59.291+0000 7ffbb4a0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-18T14:40:59.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:59.291+0000 7ffbb4a0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-18T14:40:59.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:59.291+0000 7ffbb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T14:40:59.294 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:59.291+0000 7ffbb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T14:40:59.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:59.291+0000 7ffbb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T14:40:59.295 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:40:59.292+0000 7ffbb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-18T14:41:02.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-18T14:41:02.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:41:02.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:02.272+0000 7f33a2cac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-18T14:41:02.275 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:02.272+0000 7f33a2cac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-18T14:41:02.275 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:02.272+0000 7f33a2cac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T14:41:02.275 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:02.272+0000 7f33a2cac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T14:41:02.276 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:02.272+0000 7f33a2cac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T14:41:02.276 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:02.273+0000 7f33a2cac640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-18T14:41:04.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:41:04.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:41:05.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-18T14:41:05.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:41:05.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:05.247+0000 7ff0d7ddd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-18T14:41:05.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:05.247+0000 7ff0d7ddd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-18T14:41:05.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:05.247+0000 7ff0d7ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T14:41:05.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:05.247+0000 7ff0d7ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T14:41:05.250 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:05.247+0000 7ff0d7ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T14:41:05.251 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:05.247+0000 7ff0d7ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-18T14:41:08.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-18T14:41:08.140 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-18T14:41:08.140 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:08.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-18T14:41:08.172 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:08.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-18T14:41:08.243 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:08.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-18T14:41:08.314 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:08.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-18T14:41:08.383 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:08.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-18T14:41:08.451 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:08.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-18T14:41:08.519 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:08.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-18T14:41:08.587 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:08.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-18T14:41:08.618 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:08.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-18T14:41:08.687 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:08.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-18T14:41:08.757 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:08.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-18T14:41:08.825 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:08.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-18T14:41:08.893 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:08.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-18T14:41:08.961 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:08.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-18T14:41:09.029 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:09.029 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-18T14:41:09.047 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:09.047 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-18T14:41:09.103 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:09.103 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-18T14:41:09.118 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:09.118 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-18T14:41:09.173 INFO:tasks.ceph:Adding keys to all mons... 2024-03-18T14:41:09.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-18T14:41:09.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-18T14:41:09.216 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhlGVnoDhAAblQd6YniwXUXlv3N2eZvKQ== 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhltJMFFxAAsrfhgR5dgZaElpPSyFsH7Q== 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhla1pOHhAAsDJ0P/0XWnoBakiPfEPYyA== 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBpUvhlzhYhDxAATnOtwIggkZq8Cix5xsKd7A== 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBsUvhlkHrEDRAAQHbA4maLjVfgH5XiBUVn0g== 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBvUvhlN5f9CxAAor/uSMxdAdq8E9k/Bcay0g== 2024-03-18T14:41:09.217 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQByUvhls/oKChAANwFAbqxQRefnUxXp74nxaA== 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhlM+SZExAAawGogzVd1g2EvnpMmL0r1A== 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhlQARXIxAAcOlIn0OSzieJz4X0FMie1A== 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhlADqzKhAAfG1fJKDAsis1WSLEAMQq0Q== 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB4Uvhlj6qgEhAA5pcxWwon2a8wJcS4myHh/g== 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB7Uvhlx9p2ERAAP20961lxyrshPFvUDivuhA== 2024-03-18T14:41:09.218 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB+Uvhl7I1SEBAAPkJjmK4erqFLRfqnbFkFqw== 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCBUvhldq3UDhAAgwbv2PzFpLHI4YE/Gpg/Eg== 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout:[client.2] 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhlwRCfNBAAvhl6UTpZTW7GDuq1BP+hqQ== 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout:[client.1] 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhl1tD9NxAA7VTlgLnBw8A2KI2Rro0eAQ== 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout:[client.3] 2024-03-18T14:41:09.219 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBmUvhlByUyAhAANpMa65PITql7fLcswh+vog== 2024-03-18T14:41:09.238 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhlGVnoDhAAblQd6YniwXUXlv3N2eZvKQ== 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhltJMFFxAAsrfhgR5dgZaElpPSyFsH7Q== 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhla1pOHhAAsDJ0P/0XWnoBakiPfEPYyA== 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBpUvhlzhYhDxAATnOtwIggkZq8Cix5xsKd7A== 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBsUvhlkHrEDRAAQHbA4maLjVfgH5XiBUVn0g== 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBvUvhlN5f9CxAAor/uSMxdAdq8E9k/Bcay0g== 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQByUvhls/oKChAANwFAbqxQRefnUxXp74nxaA== 2024-03-18T14:41:09.239 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhlM+SZExAAawGogzVd1g2EvnpMmL0r1A== 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhlQARXIxAAcOlIn0OSzieJz4X0FMie1A== 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhlADqzKhAAfG1fJKDAsis1WSLEAMQq0Q== 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQB4Uvhlj6qgEhAA5pcxWwon2a8wJcS4myHh/g== 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQB7Uvhlx9p2ERAAP20961lxyrshPFvUDivuhA== 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQB+Uvhl7I1SEBAAPkJjmK4erqFLRfqnbFkFqw== 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCBUvhldq3UDhAAgwbv2PzFpLHI4YE/Gpg/Eg== 2024-03-18T14:41:09.240 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-03-18T14:41:09.241 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:41:09.241 INFO:teuthology.orchestra.run.smithi092.stdout:[client.2] 2024-03-18T14:41:09.241 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhlwRCfNBAAvhl6UTpZTW7GDuq1BP+hqQ== 2024-03-18T14:41:09.241 INFO:teuthology.orchestra.run.smithi092.stdout:[client.1] 2024-03-18T14:41:09.241 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBlUvhl1tD9NxAA7VTlgLnBw8A2KI2Rro0eAQ== 2024-03-18T14:41:09.241 INFO:teuthology.orchestra.run.smithi092.stdout:[client.3] 2024-03-18T14:41:09.241 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQBmUvhlByUyAhAANpMa65PITql7fLcswh+vog== 2024-03-18T14:41:09.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T14:41:09.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T14:41:09.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T14:41:09.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T14:41:09.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T14:41:09.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.942 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:09.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T14:41:10.047 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T14:41:10.274 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-18T14:41:10.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-18T14:41:10.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T14:41:10.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-18T14:41:10.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-18T14:41:10.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T14:41:10.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-18T14:41:10.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-18T14:41:10.640 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T14:41:10.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-18T14:41:10.764 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:41:10.767 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-18T14:41:10.819 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-18T14:41:10.819 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-18T14:41:10.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-18T14:41:10.821 INFO:tasks.ceph.mon.a:Started 2024-03-18T14:41:10.822 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-18T14:41:10.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-18T14:41:10.862 INFO:tasks.ceph.mon.b:Started 2024-03-18T14:41:10.862 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-18T14:41:10.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-18T14:41:10.864 INFO:tasks.ceph.mon.c:Started 2024-03-18T14:41:10.864 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-18T14:41:10.864 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-18T14:41:10.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-18T14:41:10.866 INFO:tasks.ceph.mgr.y:Started 2024-03-18T14:41:10.866 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-18T14:41:10.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-18T14:41:10.869 INFO:tasks.ceph.mgr.x:Started 2024-03-18T14:41:10.869 INFO:tasks.ceph:Setting crush tunables to default 2024-03-18T14:41:10.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-03-18T14:41:10.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.968+0000 7fbc22ab2640 1 Processor -- start 2024-03-18T14:41:10.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.968+0000 7fbc22ab2640 1 -- start start 2024-03-18T14:41:10.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc22ab2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc1c0b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc22ab2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 0x7fbc1c0b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc22ab2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc1c15f6c0 con 0x7fbc1c143ed0 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc22ab2640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbc1c0cf7d0 con 0x7fbc1c0b0cd0 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc22ab2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c0cf590 con 0x7fbc1c1443c0 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2104f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 0x7fbc1c0b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2104f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 0x7fbc1c0b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35170/0 (socket says 172.21.15.6:35170) 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2084e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbc1c143ed0 0x7fbc1c0cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56466/0 (socket says 172.21.15.6:56466) 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2104f640 1 -- 172.21.15.6:0/1269335428 learned_addr learned my addr 172.21.15.6:0/1269335428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc1bfff640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc1c0b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2104f640 1 -- 172.21.15.6:0/1269335428 >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 msgr2=0x7fbc1c0b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2104f640 1 -- 172.21.15.6:0/1269335428 >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 msgr2=0x7fbc1c0b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2104f640 1 --2- 172.21.15.6:0/1269335428 >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 0x7fbc1c0b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc2104f640 1 --2- 172.21.15.6:0/1269335428 >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 0x7fbc1c0b3870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:41:10.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc1bfff640 1 -- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 msgr2=0x7fbc1c0b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T14:41:10.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc1bfff640 1 -- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 msgr2=0x7fbc1c0b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T14:41:10.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc1bfff640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc1c0b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T14:41:10.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:10.969+0000 7fbc1bfff640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc1c0b0790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T14:41:11.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118979337 0 0) 0x7fbc1c15f6c0 con 0x7fbc1c143ed0 2024-03-18T14:41:11.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc04005730 con 0x7fbc1c143ed0 2024-03-18T14:41:11.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 738581092 0 0) 0x7fbc04005730 con 0x7fbc1c143ed0 2024-03-18T14:41:11.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 msgr2=0x7fbc1c0b0790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T14:41:11.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc1c0b0790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 msgr2=0x7fbc1c0b3870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 --2- 172.21.15.6:0/1269335428 >> v2:172.21.15.6:3300/0 conn(0x7fbc1c0b0cd0 0x7fbc1c0b3870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c15e900 con 0x7fbc1c143ed0 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fbc0c001670 con 0x7fbc1c143ed0 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 >> v1:172.21.15.6:6789/0 conn(0x7fbc1c143ed0 legacy=0x7fbc1c0cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc04005bc0 0x7fbc04005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc04006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.158+0000 7fbc1b7fe640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc04006f50 0x7fbc040073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c0cf590 con 0x7fbc04006f50 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0400dc80 con 0x7fbc04005bc0 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0400de00 con 0x7fbc1c1443c0 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc2104f640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc04006f50 0x7fbc040073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc2084e640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc04005bc0 0x7fbc04005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc1bfff640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc04006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc2104f640 1 -- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 msgr2=0x7fbc04006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc2104f640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc1c1443c0 0x7fbc04006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc2104f640 1 -- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc04005bc0 msgr2=0x7fbc04005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc2104f640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc04005bc0 0x7fbc04005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.159+0000 7fbc2104f640 1 -- 172.21.15.6:0/1269335428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc04005730 con 0x7fbc04006f50 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.160+0000 7fbc2104f640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc04006f50 0x7fbc040073d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009cf0 tx=0x7fbc04007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d66aab369088861 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.160+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14032070 con 0x7fbc04006f50 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.160+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1402cb10 con 0x7fbc04006f50 2024-03-18T14:41:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.160+0000 7fbc1b7fe640 1 -- 172.21.15.6:0/1269335428 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc140098c0 con 0x7fbc04006f50 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.161+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1269335428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc04006f50 msgr2=0x7fbc040073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.161+0000 7fbc22ab2640 1 --2- 172.21.15.6:0/1269335428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc04006f50 0x7fbc040073d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009cf0 tx=0x7fbc04007a40 comp rx=0 tx=0).stop 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.161+0000 7fbc2104f640 1 -- 172.21.15.6:0/1269335428 reap_dead start 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.161+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1269335428 shutdown_connections 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.161+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1269335428 >> 172.21.15.6:0/1269335428 conn(0x7fbc1c06b1c0 msgr2=0x7fbc1c06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.161+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1269335428 shutdown_connections 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.161+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1269335428 wait complete. 2024-03-18T14:41:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.162+0000 7fbc22ab2640 1 Processor -- start 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.162+0000 7fbc22ab2640 1 -- start start 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc22ab2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc04006f50 0x7fbc1c138b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc22ab2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc1c06f6b0 0x7fbc1c15afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc22ab2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc1c06e340 0x7fbc1c06e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc22ab2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c15eec0 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc22ab2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c15f230 con 0x7fbc1c06e340 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc22ab2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c0b3db0 con 0x7fbc04006f50 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc1bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc1c06f6b0 0x7fbc1c15afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc1bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc1c06f6b0 0x7fbc1c15afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35192/0 (socket says 172.21.15.6:35192) 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc1bfff640 1 -- 172.21.15.6:0/1151694959 learned_addr learned my addr 172.21.15.6:0/1151694959 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc2104f640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc1c06e340 0x7fbc1c06e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.163+0000 7fbc2084e640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc04006f50 0x7fbc1c138b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc1bfff640 1 -- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc04006f50 msgr2=0x7fbc1c138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc1bfff640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc04006f50 0x7fbc1c138b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc1bfff640 1 -- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc1c06e340 msgr2=0x7fbc1c06e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc1bfff640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc1c06e340 0x7fbc1c06e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc1bfff640 1 -- 172.21.15.6:0/1151694959 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c15dc10 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc1bfff640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc1c06f6b0 0x7fbc1c15afc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc100108a0 tx=0x7fbc10010d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc197fa640 1 -- 172.21.15.6:0/1151694959 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1001a020 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc197fa640 1 -- 172.21.15.6:0/1151694959 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1000e9c0 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc197fa640 1 -- 172.21.15.6:0/1151694959 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000edb0 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c15b8c0 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c15be80 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc197fa640 1 -- 172.21.15.6:0/1151694959 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbc1001d020 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.164+0000 7fbc197fa640 1 -- 172.21.15.6:0/1151694959 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbc100148e0 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.165+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c0751b0 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.167+0000 7fbc197fa640 1 -- 172.21.15.6:0/1151694959 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbc10024020 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.233+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbc1c0cf160 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.235+0000 7fbc197fa640 1 -- 172.21.15.6:0/1151694959 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fbc10014430 con 0x7fbc1c06f6b0 2024-03-18T14:41:11.237 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-03-18T14:41:11.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.235+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc1c06f6b0 msgr2=0x7fbc1c15afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.235+0000 7fbc22ab2640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc1c06f6b0 0x7fbc1c15afc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc100108a0 tx=0x7fbc10010d60 comp rx=0 tx=0).stop 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.236+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 shutdown_connections 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.236+0000 7fbc22ab2640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc1c06e340 0x7fbc1c06e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.236+0000 7fbc22ab2640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc1c06f6b0 0x7fbc1c15afc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.236+0000 7fbc22ab2640 1 --2- 172.21.15.6:0/1151694959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc04006f50 0x7fbc1c138b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.236+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 >> 172.21.15.6:0/1151694959 conn(0x7fbc1c06b1c0 msgr2=0x7fbc1c072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.236+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 shutdown_connections 2024-03-18T14:41:11.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:11.236+0000 7fbc22ab2640 1 -- 172.21.15.6:0/1151694959 wait complete. 2024-03-18T14:41:11.249 INFO:tasks.ceph:check_enable_crimson: False 2024-03-18T14:41:11.249 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-18T14:41:11.249 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:11.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-18T14:41:11.275 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:11.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-18T14:41:11.342 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:11.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-18T14:41:11.408 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:11.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-18T14:41:11.474 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:11.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-18T14:41:11.502 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:11.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-18T14:41:11.568 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:11.568 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-18T14:41:11.618 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:11.616+0000 7fa6eb849040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:41:11.635 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:41:11.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-18T14:41:11.674 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:11.672+0000 7fa6eb849040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:41:11.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 297ee578-07f6-413d-94d4-1feabcab858f 0 2024-03-18T14:41:11.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.839+0000 7f496253b640 1 Processor -- start 2024-03-18T14:41:11.841 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.839+0000 7f496253b640 1 -- start start 2024-03-18T14:41:11.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f496253b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f495c0c4040 0x7f495c0cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f496253b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c0ce290 con 0x7f495c15bc60 2024-03-18T14:41:11.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f496253b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c0cd9f0 con 0x7f495c06b8d0 2024-03-18T14:41:11.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f496253b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f495c0b0200 con 0x7f495c0c4040 2024-03-18T14:41:11.842 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f4960ad8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f495c0c4040 0x7f495c0cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f495b7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f495c06b8d0 0x7f495c0b0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:42696/0 (socket says 172.21.15.92:42696) 2024-03-18T14:41:11.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f495b7fe640 1 -- 172.21.15.92:0/3753186846 learned_addr learned my addr 172.21.15.92:0/3753186846 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-18T14:41:11.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f495affd640 1 -- 172.21.15.92:0/3753186846 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1324808971 0 0) 0x7f495c0cd9f0 con 0x7f495c06b8d0 2024-03-18T14:41:11.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f495affd640 1 -- 172.21.15.92:0/3753186846 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4944003520 con 0x7f495c06b8d0 2024-03-18T14:41:11.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f4960ad8640 1 -- 172.21.15.92:0/3753186846 >> v1:172.21.15.92:6789/0 conn(0x7f495c06b8d0 legacy=0x7f495c0b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f4960ad8640 1 -- 172.21.15.92:0/3753186846 >> v1:172.21.15.6:6789/0 conn(0x7f495c15bc60 legacy=0x7f495c06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.843 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.840+0000 7f4960ad8640 1 -- 172.21.15.92:0/3753186846 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495c0cdcd0 con 0x7f495c0c4040 2024-03-18T14:41:11.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.841+0000 7f4960ad8640 1 --2- 172.21.15.92:0/3753186846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c0c4040 0x7f495c0cc660 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4954009d90 tx=0x7f495402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2383ff3ecfc56b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:11.844 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.841+0000 7f495affd640 1 -- 172.21.15.92:0/3753186846 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954039070 con 0x7f495c0c4040 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.841+0000 7f495affd640 1 -- 172.21.15.92:0/3753186846 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f495402cc60 con 0x7f495c0c4040 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.841+0000 7f495affd640 1 -- 172.21.15.92:0/3753186846 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954034660 con 0x7f495c0c4040 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.842+0000 7f496253b640 1 -- 172.21.15.92:0/3753186846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c0c4040 msgr2=0x7f495c0cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.842+0000 7f496253b640 1 --2- 172.21.15.92:0/3753186846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c0c4040 0x7f495c0cc660 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4954009d90 tx=0x7f495402beb0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.842+0000 7f496253b640 1 -- 172.21.15.92:0/3753186846 shutdown_connections 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.842+0000 7f496253b640 1 --2- 172.21.15.92:0/3753186846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c0c4040 0x7f495c0cc660 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.842+0000 7f496253b640 1 -- 172.21.15.92:0/3753186846 >> 172.21.15.92:0/3753186846 conn(0x7f495c0d09b0 msgr2=0x7f495c0d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.842+0000 7f496253b640 1 -- 172.21.15.92:0/3753186846 shutdown_connections 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.842+0000 7f496253b640 1 -- 172.21.15.92:0/3753186846 wait complete. 2024-03-18T14:41:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.843+0000 7f496253b640 1 Processor -- start 2024-03-18T14:41:11.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.843+0000 7f496253b640 1 -- start start 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f496253b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c06b8d0 0x7f495c155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f496253b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f495c0c4040 0x7f495c0b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f496253b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f495c15bc60 0x7f495c15e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f496253b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f495c0cd650 con 0x7f495c15bc60 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f496253b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f495c15c040 con 0x7f495c06b8d0 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f496253b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f495c0cd7d0 con 0x7f495c0c4040 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495b7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f495c0c4040 0x7f495c0b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c06b8d0 0x7f495c155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495b7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f495c0c4040 0x7f495c0b6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:45340/0 (socket says 172.21.15.92:45340) 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c06b8d0 0x7f495c155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58878/0 (socket says 172.21.15.92:58878) 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495b7fe640 1 -- 172.21.15.92:0/4214923471 learned_addr learned my addr 172.21.15.92:0/4214923471 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f4960ad8640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f495c15bc60 0x7f495c15e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 -- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f495c0c4040 msgr2=0x7f495c0b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.848 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f495c0c4040 0x7f495c0b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 -- 172.21.15.92:0/4214923471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f495c15bc60 msgr2=0x7f495c15e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f495c15bc60 0x7f495c15e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 -- 172.21.15.92:0/4214923471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495c06fd80 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495b7fe640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f495c0c4040 0x7f495c0b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f495bfff640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c06b8d0 0x7f495c155c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f495000c340 tx=0x7f495000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f4958ff9640 1 -- 172.21.15.92:0/4214923471 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49500052d0 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.844+0000 7f4958ff9640 1 -- 172.21.15.92:0/4214923471 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4950005490 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.845+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495c15dcb0 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.845+0000 7f4958ff9640 1 -- 172.21.15.92:0/4214923471 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950007260 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.845+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f495c070300 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.845+0000 7f4958ff9640 1 -- 172.21.15.92:0/4214923471 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4950007400 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.845+0000 7f4958ff9640 1 -- 172.21.15.92:0/4214923471 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f4950003a90 con 0x7f495c06b8d0 2024-03-18T14:41:11.849 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.845+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f495c0751b0 con 0x7f495c06b8d0 2024-03-18T14:41:11.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.848+0000 7f4958ff9640 1 -- 172.21.15.92:0/4214923471 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f495c0ce290 con 0x7f495c06b8d0 2024-03-18T14:41:11.921 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.919+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "297ee578-07f6-413d-94d4-1feabcab858f", "id": 0} v 0) -- 0x7f495c06b260 con 0x7f495c06b8d0 2024-03-18T14:41:11.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.922+0000 7f4958ff9640 1 -- 172.21.15.92:0/4214923471 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "297ee578-07f6-413d-94d4-1feabcab858f", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f4950003c90 con 0x7f495c06b8d0 2024-03-18T14:41:11.924 INFO:teuthology.orchestra.run.smithi092.stdout:0 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c06b8d0 msgr2=0x7f495c155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c06b8d0 0x7f495c155c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f495000c340 tx=0x7f495000c800 comp rx=0 tx=0).stop 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 shutdown_connections 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f495c15bc60 0x7f495c15e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f495c0c4040 0x7f495c0b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 --2- 172.21.15.92:0/4214923471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f495c06b8d0 0x7f495c155c50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 >> 172.21.15.92:0/4214923471 conn(0x7f495c0d09b0 msgr2=0x7f495c156c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 shutdown_connections 2024-03-18T14:41:11.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:11.923+0000 7f496253b640 1 -- 172.21.15.92:0/4214923471 wait complete. 2024-03-18T14:41:11.935 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 0517bd2c-8330-4ffb-9920-e4d01560c349 1 2024-03-18T14:41:12.014 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.011+0000 7f7d79414040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T14:41:12.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.033+0000 7fce31ddf640 1 Processor -- start 2024-03-18T14:41:12.036 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.033+0000 7fce31ddf640 1 -- start start 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce31ddf640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce2c0cf6a0 0x7fce2c0ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce31ddf640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce2c0d14d0 0x7fce2c0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce31ddf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2c0b2280 con 0x7fce2c0b1ea0 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce31ddf640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fce2c0ceb60 con 0x7fce2c0d14d0 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce31ddf640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fce2c15b3e0 con 0x7fce2c0cf6a0 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce2affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce2c0cf6a0 0x7fce2c0ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce2affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce2c0cf6a0 0x7fce2c0ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58894/0 (socket says 172.21.15.92:58894) 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce2affd640 1 -- 172.21.15.92:0/4097140095 learned_addr learned my addr 172.21.15.92:0/4097140095 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.034+0000 7fce2bfff640 1 --2- 172.21.15.92:0/4097140095 >> v2:172.21.15.6:3300/0 conn(0x7fce2c0d14d0 0x7fce2c0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.035+0000 7fce2affd640 1 -- 172.21.15.92:0/4097140095 >> v2:172.21.15.6:3300/0 conn(0x7fce2c0d14d0 msgr2=0x7fce2c0cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:12.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.035+0000 7fce2affd640 1 --2- 172.21.15.92:0/4097140095 >> v2:172.21.15.6:3300/0 conn(0x7fce2c0d14d0 0x7fce2c0cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.035+0000 7fce2affd640 1 -- 172.21.15.92:0/4097140095 >> v1:172.21.15.6:6789/0 conn(0x7fce2c0b1ea0 legacy=0x7fce2c0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.038 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.035+0000 7fce2affd640 1 -- 172.21.15.92:0/4097140095 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2c15b6c0 con 0x7fce2c0cf6a0 2024-03-18T14:41:12.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.035+0000 7fce2affd640 1 --2- 172.21.15.92:0/4097140095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0cf6a0 0x7fce2c0ce580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0c7a10 tx=0x7fce2002e170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88a1d061cc6a7860 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce2a7fc640 1 -- 172.21.15.92:0/4097140095 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2003b070 con 0x7fce2c0cf6a0 2024-03-18T14:41:12.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce2a7fc640 1 -- 172.21.15.92:0/4097140095 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce20036440 con 0x7fce2c0cf6a0 2024-03-18T14:41:12.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce2a7fc640 1 -- 172.21.15.92:0/4097140095 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce20036810 con 0x7fce2c0cf6a0 2024-03-18T14:41:12.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 -- 172.21.15.92:0/4097140095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0cf6a0 msgr2=0x7fce2c0ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.039 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 --2- 172.21.15.92:0/4097140095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0cf6a0 0x7fce2c0ce580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0c7a10 tx=0x7fce2002e170 comp rx=0 tx=0).stop 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 -- 172.21.15.92:0/4097140095 shutdown_connections 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 --2- 172.21.15.92:0/4097140095 >> v2:172.21.15.6:3300/0 conn(0x7fce2c0d14d0 0x7fce2c0cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 --2- 172.21.15.92:0/4097140095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0cf6a0 0x7fce2c0ce580 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 -- 172.21.15.92:0/4097140095 >> 172.21.15.92:0/4097140095 conn(0x7fce2c0d09b0 msgr2=0x7fce2c0d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 -- 172.21.15.92:0/4097140095 shutdown_connections 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.036+0000 7fce31ddf640 1 -- 172.21.15.92:0/4097140095 wait complete. 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.037+0000 7fce31ddf640 1 Processor -- start 2024-03-18T14:41:12.040 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.037+0000 7fce31ddf640 1 -- start start 2024-03-18T14:41:12.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce31ddf640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0b1ea0 0x7fce2c09a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce31ddf640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce2c0cf6a0 0x7fce2c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce31ddf640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce2c0d14d0 0x7fce2c0b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce31ddf640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fce2c15b0d0 con 0x7fce2c0cf6a0 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce31ddf640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fce2c0cdcf0 con 0x7fce2c0b1ea0 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce31ddf640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fce2c15bec0 con 0x7fce2c0d14d0 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce2b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0b1ea0 0x7fce2c09a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce2bfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce2c0d14d0 0x7fce2c0b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce2b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0b1ea0 0x7fce2c09a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58896/0 (socket says 172.21.15.92:58896) 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce2b7fe640 1 -- 172.21.15.92:0/3207052465 learned_addr learned my addr 172.21.15.92:0/3207052465 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.038+0000 7fce2affd640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce2c0cf6a0 0x7fce2c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce2bfff640 1 -- 172.21.15.92:0/3207052465 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0b1ea0 msgr2=0x7fce2c09a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce2bfff640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0b1ea0 0x7fce2c09a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce2bfff640 1 -- 172.21.15.92:0/3207052465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce2c0cf6a0 msgr2=0x7fce2c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce2bfff640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce2c0cf6a0 0x7fce2c0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce2bfff640 1 -- 172.21.15.92:0/3207052465 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2c09d500 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce2bfff640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce2c0d14d0 0x7fce2c0b45b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fce2400eba0 tx=0x7fce2400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce13fff640 1 -- 172.21.15.92:0/3207052465 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24018020 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce13fff640 1 -- 172.21.15.92:0/3207052465 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce2400cd20 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce2c09d7e0 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce13fff640 1 -- 172.21.15.92:0/3207052465 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce240146c0 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce2c1e39f0 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce13fff640 1 -- 172.21.15.92:0/3207052465 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fce24005d10 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.039+0000 7fce13fff640 1 -- 172.21.15.92:0/3207052465 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fce24013790 con 0x7fce2c0d14d0 2024-03-18T14:41:12.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.040+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce2c0751b0 con 0x7fce2c0d14d0 2024-03-18T14:41:12.045 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.042+0000 7fce13fff640 1 -- 172.21.15.92:0/3207052465 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fce24022030 con 0x7fce2c0d14d0 2024-03-18T14:41:12.070 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.067+0000 7f7d79414040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T14:41:12.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.140+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0517bd2c-8330-4ffb-9920-e4d01560c349", "id": 1} v 0) -- 0x7fce2c0b1a10 con 0x7fce2c0d14d0 2024-03-18T14:41:12.145 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.144+0000 7fce13fff640 1 -- 172.21.15.92:0/3207052465 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0517bd2c-8330-4ffb-9920-e4d01560c349", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fce24010040 con 0x7fce2c0d14d0 2024-03-18T14:41:12.146 INFO:teuthology.orchestra.run.smithi092.stdout:1 2024-03-18T14:41:12.146 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce2c0d14d0 msgr2=0x7fce2c0b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce2c0d14d0 0x7fce2c0b45b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fce2400eba0 tx=0x7fce2400c490 comp rx=0 tx=0).stop 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 shutdown_connections 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce2c0d14d0 0x7fce2c0b45b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce2c0cf6a0 0x7fce2c0c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 --2- 172.21.15.92:0/3207052465 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce2c0b1ea0 0x7fce2c09a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 >> 172.21.15.92:0/3207052465 conn(0x7fce2c0d09b0 msgr2=0x7fce2c0c7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 shutdown_connections 2024-03-18T14:41:12.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.145+0000 7fce31ddf640 1 -- 172.21.15.92:0/3207052465 wait complete. 2024-03-18T14:41:12.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new b840e6ce-6fc9-4428-a13c-e45ee2431bfd 2 2024-03-18T14:41:12.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.256+0000 7f7ed71b2640 1 Processor -- start 2024-03-18T14:41:12.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.256+0000 7f7ed71b2640 1 -- start start 2024-03-18T14:41:12.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.256+0000 7f7ed71b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00ce9d0 0x7f7ed00cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.256+0000 7f7ed71b2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed00d2970 con 0x7f7ed00d0730 2024-03-18T14:41:12.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.256+0000 7f7ed71b2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed00cfac0 con 0x7f7ed00d2510 2024-03-18T14:41:12.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.256+0000 7f7ed71b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed00cd8d0 con 0x7f7ed00ce9d0 2024-03-18T14:41:12.261 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.257+0000 7f7ed4f4e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00ce9d0 0x7f7ed00cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.257+0000 7f7ecffff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7ed00d2510 0x7f7ed00d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:42702/0 (socket says 172.21.15.92:42702) 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.257+0000 7f7ecffff640 1 -- 172.21.15.92:0/3786670884 learned_addr learned my addr 172.21.15.92:0/3786670884 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.257+0000 7f7ed4f4e640 1 -- 172.21.15.92:0/3786670884 >> v1:172.21.15.92:6789/0 conn(0x7f7ed00d2510 legacy=0x7f7ed00d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.257+0000 7f7ed4f4e640 1 -- 172.21.15.92:0/3786670884 >> v1:172.21.15.6:6789/0 conn(0x7f7ed00d0730 legacy=0x7f7ed015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.257+0000 7f7ecf7fe640 1 -- 172.21.15.92:0/3786670884 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1562000591 0 0) 0x7f7ed00cfac0 con 0x7f7ed00d2510 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.257+0000 7f7ed4f4e640 1 -- 172.21.15.92:0/3786670884 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed00cccc0 con 0x7f7ed00ce9d0 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed4f4e640 1 --2- 172.21.15.92:0/3786670884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00ce9d0 0x7f7ed00cc650 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40097c0 tx=0x7f7ec402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88504a80fd57e3a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ecf7fe640 1 -- 172.21.15.92:0/3786670884 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4035020 con 0x7f7ed00ce9d0 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ecf7fe640 1 -- 172.21.15.92:0/3786670884 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec4031800 con 0x7f7ed00ce9d0 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ecf7fe640 1 -- 172.21.15.92:0/3786670884 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4031bd0 con 0x7f7ed00ce9d0 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed71b2640 1 -- 172.21.15.92:0/3786670884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00ce9d0 msgr2=0x7f7ed00cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed71b2640 1 --2- 172.21.15.92:0/3786670884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00ce9d0 0x7f7ed00cc650 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40097c0 tx=0x7f7ec402b8a0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed71b2640 1 -- 172.21.15.92:0/3786670884 shutdown_connections 2024-03-18T14:41:12.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed71b2640 1 --2- 172.21.15.92:0/3786670884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00ce9d0 0x7f7ed00cc650 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed71b2640 1 -- 172.21.15.92:0/3786670884 >> 172.21.15.92:0/3786670884 conn(0x7f7ed008b0c0 msgr2=0x7f7ed00d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed71b2640 1 -- 172.21.15.92:0/3786670884 shutdown_connections 2024-03-18T14:41:12.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.258+0000 7f7ed71b2640 1 -- 172.21.15.92:0/3786670884 wait complete. 2024-03-18T14:41:12.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 Processor -- start 2024-03-18T14:41:12.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 -- start start 2024-03-18T14:41:12.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed00ce9d0 0x7f7ed01504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00d0730 0x7f7ed009ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ed00d2510 0x7f7ed009cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed00cd560 con 0x7f7ed00d2510 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed00cd280 con 0x7f7ed00ce9d0 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.260+0000 7f7ed71b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed00cedb0 con 0x7f7ed00d0730 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00d0730 0x7f7ed009ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ed4f4e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed00ce9d0 0x7f7ed01504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00d0730 0x7f7ed009ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:45364/0 (socket says 172.21.15.92:45364) 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 -- 172.21.15.92:0/2280915486 learned_addr learned my addr 172.21.15.92:0/2280915486 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ed574f640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ed00d2510 0x7f7ed009cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 -- 172.21.15.92:0/2280915486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed00ce9d0 msgr2=0x7f7ed01504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed00ce9d0 0x7f7ed01504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 -- 172.21.15.92:0/2280915486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ed00d2510 msgr2=0x7f7ed009cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ed00d2510 0x7f7ed009cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 -- 172.21.15.92:0/2280915486 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed014f420 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecffff640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00d0730 0x7f7ed009ef70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00c3c0 tx=0x7f7ebc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecd7fa640 1 -- 172.21.15.92:0/2280915486 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc017070 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecd7fa640 1 -- 172.21.15.92:0/2280915486 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ebc005ce0 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecd7fa640 1 -- 172.21.15.92:0/2280915486 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc007310 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed014f700 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed014fcc0 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecd7fa640 1 -- 172.21.15.92:0/2280915486 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7ebc01a070 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.261+0000 7f7ecd7fa640 1 -- 172.21.15.92:0/2280915486 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f7ebc012820 con 0x7f7ed00d0730 2024-03-18T14:41:12.267 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.262+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed00751b0 con 0x7f7ed00d0730 2024-03-18T14:41:12.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.266+0000 7f7ecd7fa640 1 -- 172.21.15.92:0/2280915486 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7ebc021070 con 0x7f7ed00d0730 2024-03-18T14:41:12.272 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.270+0000 7fa6eb849040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:41:12.328 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.326+0000 7fa6eb849040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:41:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.339+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b840e6ce-6fc9-4428-a13c-e45ee2431bfd", "id": 2} v 0) -- 0x7f7ed00cfe00 con 0x7f7ed00d0730 2024-03-18T14:41:12.346 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.344+0000 7f7ecd7fa640 1 -- 172.21.15.92:0/2280915486 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b840e6ce-6fc9-4428-a13c-e45ee2431bfd", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f7ebc0125b0 con 0x7f7ed00d0730 2024-03-18T14:41:12.346 INFO:teuthology.orchestra.run.smithi092.stdout:2 2024-03-18T14:41:12.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00d0730 msgr2=0x7f7ed009ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00d0730 0x7f7ed009ef70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00c3c0 tx=0x7f7ebc00c880 comp rx=0 tx=0).stop 2024-03-18T14:41:12.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 shutdown_connections 2024-03-18T14:41:12.347 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ed00d2510 0x7f7ed009cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ed00d0730 0x7f7ed009ef70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 --2- 172.21.15.92:0/2280915486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed00ce9d0 0x7f7ed01504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 >> 172.21.15.92:0/2280915486 conn(0x7f7ed008b0c0 msgr2=0x7f7ed00b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 shutdown_connections 2024-03-18T14:41:12.348 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.345+0000 7f7ed71b2640 1 -- 172.21.15.92:0/2280915486 wait complete. 2024-03-18T14:41:12.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 2cf1864e-77e7-4115-b871-12395a882274 3 2024-03-18T14:41:12.376 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.374+0000 7fa6eb849040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:41:12.433 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.431+0000 7fa6eb849040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:41:12.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.455+0000 7f143a281640 1 Processor -- start 2024-03-18T14:41:12.457 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.455+0000 7f143a281640 1 -- start start 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.455+0000 7f143a281640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143415be60 0x7f14340ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f143a281640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14340cf930 0x7f14340d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f143a281640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f143406fcc0 con 0x7f14340d0670 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f143a281640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f143406ffa0 con 0x7f14340cf930 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f143a281640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1434070280 con 0x7f143415be60 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f14337fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14340cf930 0x7f14340d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f1433fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143415be60 0x7f14340ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f14337fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14340cf930 0x7f14340d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54178/0 (socket says 172.21.15.92:54178) 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f1433fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143415be60 0x7f14340ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:51526/0 (socket says 172.21.15.92:51526) 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f14337fe640 1 -- 172.21.15.92:0/1200884791 learned_addr learned my addr 172.21.15.92:0/1200884791 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f1433fff640 1 -- 172.21.15.92:0/1200884791 >> v2:172.21.15.92:3300/0 conn(0x7f14340cf930 msgr2=0x7f14340d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f1433fff640 1 --2- 172.21.15.92:0/1200884791 >> v2:172.21.15.92:3300/0 conn(0x7f14340cf930 0x7f14340d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.460 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f1433fff640 1 -- 172.21.15.92:0/1200884791 >> v1:172.21.15.6:6789/0 conn(0x7f14340d0670 legacy=0x7f14340cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f1433fff640 1 -- 172.21.15.92:0/1200884791 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1434070560 con 0x7f143415be60 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.456+0000 7f14337fe640 1 --2- 172.21.15.92:0/1200884791 >> v2:172.21.15.92:3300/0 conn(0x7f14340cf930 0x7f14340d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f1433fff640 1 --2- 172.21.15.92:0/1200884791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143415be60 0x7f14340ceb90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f14240097a0 tx=0x7f142402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddc0dc574a83ebbf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f1432ffd640 1 -- 172.21.15.92:0/1200884791 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1424031780 con 0x7f143415be60 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f1432ffd640 1 -- 172.21.15.92:0/1200884791 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1424037070 con 0x7f143415be60 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f1432ffd640 1 -- 172.21.15.92:0/1200884791 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1424031b40 con 0x7f143415be60 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f143a281640 1 -- 172.21.15.92:0/1200884791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143415be60 msgr2=0x7f14340ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f143a281640 1 --2- 172.21.15.92:0/1200884791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143415be60 0x7f14340ceb90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f14240097a0 tx=0x7f142402b820 comp rx=0 tx=0).stop 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f143a281640 1 -- 172.21.15.92:0/1200884791 shutdown_connections 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f143a281640 1 --2- 172.21.15.92:0/1200884791 >> v2:172.21.15.92:3300/0 conn(0x7f14340cf930 0x7f14340d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f143a281640 1 --2- 172.21.15.92:0/1200884791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143415be60 0x7f14340ceb90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.457+0000 7f143a281640 1 -- 172.21.15.92:0/1200884791 >> 172.21.15.92:0/1200884791 conn(0x7f14340d1b50 msgr2=0x7f14340d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.458+0000 7f143a281640 1 -- 172.21.15.92:0/1200884791 shutdown_connections 2024-03-18T14:41:12.461 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.458+0000 7f143a281640 1 -- 172.21.15.92:0/1200884791 wait complete. 2024-03-18T14:41:12.464 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.459+0000 7f143a281640 1 Processor -- start 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.459+0000 7f143a281640 1 -- start start 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143a281640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14340cf930 0x7f14341f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143a281640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14340d0670 0x7f14340b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143a281640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f143415be60 0x7f143415ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143a281640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f14340ae710 con 0x7f143415be60 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143a281640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f143415c240 con 0x7f14340d0670 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143a281640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f14340ae890 con 0x7f14340cf930 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14340d0670 0x7f14340b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14340d0670 0x7f14340b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54184/0 (socket says 172.21.15.92:54184) 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 -- 172.21.15.92:0/1288062737 learned_addr learned my addr 172.21.15.92:0/1288062737 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f1433fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14340cf930 0x7f14341f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.465 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143881e640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f143415be60 0x7f143415ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 -- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14340cf930 msgr2=0x7f14341f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14340cf930 0x7f14341f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 -- 172.21.15.92:0/1288062737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f143415be60 msgr2=0x7f143415ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f143415be60 0x7f143415ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 -- 172.21.15.92:0/1288062737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14341f5580 con 0x7f14340d0670 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f1433fff640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14340cf930 0x7f14341f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f14337fe640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14340d0670 0x7f14340b2c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f142800e9b0 tx=0x7f142800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f1430ff9640 1 -- 172.21.15.92:0/1288062737 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428017070 con 0x7f14340d0670 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f1430ff9640 1 -- 172.21.15.92:0/1288062737 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f142800c9a0 con 0x7f14340d0670 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f1430ff9640 1 -- 172.21.15.92:0/1288062737 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142800cd50 con 0x7f14340d0670 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.460+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14341f5860 con 0x7f14340d0670 2024-03-18T14:41:12.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.461+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14341f5e20 con 0x7f14340d0670 2024-03-18T14:41:12.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.461+0000 7f1430ff9640 1 -- 172.21.15.92:0/1288062737 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f142801a070 con 0x7f14340d0670 2024-03-18T14:41:12.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.461+0000 7f1430ff9640 1 -- 172.21.15.92:0/1288062737 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f1428013470 con 0x7f14340d0670 2024-03-18T14:41:12.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.462+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14340751b0 con 0x7f14340d0670 2024-03-18T14:41:12.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.464+0000 7f1430ff9640 1 -- 172.21.15.92:0/1288062737 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1428006020 con 0x7f14340d0670 2024-03-18T14:41:12.478 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.476+0000 7fa6eb849040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:41:12.537 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.536+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2cf1864e-77e7-4115-b871-12395a882274", "id": 3} v 0) -- 0x7f143415d5d0 con 0x7f14340d0670 2024-03-18T14:41:12.542 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.541+0000 7f1430ff9640 1 -- 172.21.15.92:0/1288062737 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2cf1864e-77e7-4115-b871-12395a882274", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f14280064d0 con 0x7f14340d0670 2024-03-18T14:41:12.543 INFO:teuthology.orchestra.run.smithi092.stdout:3 2024-03-18T14:41:12.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14340d0670 msgr2=0x7f14340b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14340d0670 0x7f14340b2c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f142800e9b0 tx=0x7f142800ee70 comp rx=0 tx=0).stop 2024-03-18T14:41:12.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 shutdown_connections 2024-03-18T14:41:12.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f143415be60 0x7f143415ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14340d0670 0x7f14340b2c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 --2- 172.21.15.92:0/1288062737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14340cf930 0x7f14341f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 >> 172.21.15.92:0/1288062737 conn(0x7f14340d1b50 msgr2=0x7f14340b6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 shutdown_connections 2024-03-18T14:41:12.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.542+0000 7f143a281640 1 -- 172.21.15.92:0/1288062737 wait complete. 2024-03-18T14:41:12.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new ad5dd4ac-44f8-4929-807d-244709ff1a10 4 2024-03-18T14:41:12.598 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.596+0000 7fa6eb849040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:41:12.618 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.615+0000 7f7d79414040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T14:41:12.671 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.669+0000 7fa6eb849040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:41:12.674 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.671+0000 7f7d79414040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T14:41:12.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.692+0000 7fc9bc64b640 1 Processor -- start 2024-03-18T14:41:12.695 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.693+0000 7fc9bc64b640 1 -- start start 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9bc64b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc9b4067330 0x7fc9b415eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9bc64b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b40ad080 con 0x7fc9b40b2340 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9bc64b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b40ad360 con 0x7fc9b415e000 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9bc64b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc9b40ad640 con 0x7fc9b4067330 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9babe8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc9b4067330 0x7fc9b415eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9ba3e7640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc9b415e000 0x7fc9b40b1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57396/0 (socket says 172.21.15.92:57396) 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9babe8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc9b4067330 0x7fc9b415eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54196/0 (socket says 172.21.15.92:54196) 2024-03-18T14:41:12.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9ba3e7640 1 -- 172.21.15.92:0/3162761510 learned_addr learned my addr 172.21.15.92:0/3162761510 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-18T14:41:12.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9babe8640 1 -- 172.21.15.92:0/3162761510 >> v1:172.21.15.92:6789/0 conn(0x7fc9b415e000 legacy=0x7fc9b40b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9babe8640 1 -- 172.21.15.92:0/3162761510 >> v1:172.21.15.6:6789/0 conn(0x7fc9b40b2340 legacy=0x7fc9b415e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.697 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.694+0000 7fc9babe8640 1 -- 172.21.15.92:0/3162761510 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b40ad920 con 0x7fc9b4067330 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.695+0000 7fc9babe8640 1 --2- 172.21.15.92:0/3162761510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415eaf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009d90 tx=0x7fc9ac02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75da0b5fa4dae93c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.695+0000 7fc9b93e5640 1 -- 172.21.15.92:0/3162761510 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac039070 con 0x7fc9b4067330 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.695+0000 7fc9b93e5640 1 -- 172.21.15.92:0/3162761510 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9ac02cc60 con 0x7fc9b4067330 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.695+0000 7fc9b93e5640 1 -- 172.21.15.92:0/3162761510 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac034660 con 0x7fc9b4067330 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.696+0000 7fc9bc64b640 1 -- 172.21.15.92:0/3162761510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 msgr2=0x7fc9b415eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.696+0000 7fc9bc64b640 1 --2- 172.21.15.92:0/3162761510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415eaf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009d90 tx=0x7fc9ac02beb0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.696+0000 7fc9bc64b640 1 -- 172.21.15.92:0/3162761510 shutdown_connections 2024-03-18T14:41:12.698 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.696+0000 7fc9bc64b640 1 --2- 172.21.15.92:0/3162761510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415eaf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.699 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.696+0000 7fc9bc64b640 1 -- 172.21.15.92:0/3162761510 >> 172.21.15.92:0/3162761510 conn(0x7fc9b40ccc30 msgr2=0x7fc9b40cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.699 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.696+0000 7fc9bc64b640 1 -- 172.21.15.92:0/3162761510 shutdown_connections 2024-03-18T14:41:12.699 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.696+0000 7fc9bc64b640 1 -- 172.21.15.92:0/3162761510 wait complete. 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.697+0000 7fc9bc64b640 1 Processor -- start 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.697+0000 7fc9bc64b640 1 -- start start 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9b40b2340 0x7fc9b41e6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9b415e000 0x7fc9b41e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b40cd8a0 con 0x7fc9b40b2340 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b40cd720 con 0x7fc9b4067330 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b40ae3a0 con 0x7fc9b415e000 2024-03-18T14:41:12.701 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9babe8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9b415e000 0x7fc9b41e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54208/0 (socket says 172.21.15.92:54208) 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 -- 172.21.15.92:0/2600871764 learned_addr learned my addr 172.21.15.92:0/2600871764 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9b9be6640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9b40b2340 0x7fc9b41e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 -- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9b415e000 msgr2=0x7fc9b41e6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9b415e000 0x7fc9b41e6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 -- 172.21.15.92:0/2600871764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9b40b2340 msgr2=0x7fc9b41e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9b40b2340 0x7fc9b41e6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 -- 172.21.15.92:0/2600871764 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b41e47f0 con 0x7fc9b4067330 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9babe8640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9b415e000 0x7fc9b41e6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9ba3e7640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415a340 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c480 tx=0x7fc9b000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.702 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9aaffd640 1 -- 172.21.15.92:0/2600871764 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0017070 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9aaffd640 1 -- 172.21.15.92:0/2600871764 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b0005ce0 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b41e4ad0 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9aaffd640 1 -- 172.21.15.92:0/2600871764 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b00072f0 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.698+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b41e5090 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.699+0000 7fc9aaffd640 1 -- 172.21.15.92:0/2600871764 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc9b001a070 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.699+0000 7fc9aaffd640 1 -- 172.21.15.92:0/2600871764 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fc9b001ccb0 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.699+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b4068420 con 0x7fc9b4067330 2024-03-18T14:41:12.703 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.701+0000 7fc9aaffd640 1 -- 172.21.15.92:0/2600871764 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc9b00074e0 con 0x7fc9b4067330 2024-03-18T14:41:12.720 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.718+0000 7fa6eb849040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:41:12.720 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.718+0000 7f7d79414040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T14:41:12.774 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.772+0000 7fa6eb849040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:41:12.775 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.774+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ad5dd4ac-44f8-4929-807d-244709ff1a10", "id": 4} v 0) -- 0x7fc9b41e7040 con 0x7fc9b4067330 2024-03-18T14:41:12.776 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.773+0000 7f7d79414040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T14:41:12.780 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.779+0000 7fc9aaffd640 1 -- 172.21.15.92:0/2600871764 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad5dd4ac-44f8-4929-807d-244709ff1a10", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fc9b001c440 con 0x7fc9b4067330 2024-03-18T14:41:12.781 INFO:teuthology.orchestra.run.smithi092.stdout:4 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 msgr2=0x7fc9b415a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415a340 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c480 tx=0x7fc9b000c940 comp rx=0 tx=0).stop 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 shutdown_connections 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9b415e000 0x7fc9b41e6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9b40b2340 0x7fc9b41e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 --2- 172.21.15.92:0/2600871764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9b4067330 0x7fc9b415a340 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 >> 172.21.15.92:0/2600871764 conn(0x7fc9b40ccc30 msgr2=0x7fc9b406f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 shutdown_connections 2024-03-18T14:41:12.782 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.780+0000 7fc9bc64b640 1 -- 172.21.15.92:0/2600871764 wait complete. 2024-03-18T14:41:12.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 7390e35b-e1e4-4b0d-87da-923c0952b58c 5 2024-03-18T14:41:12.821 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.818+0000 7f7d79414040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T14:41:12.824 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:12.822+0000 7fa6eb849040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:41:12.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.923+0000 7fdb11f60640 1 Processor -- start 2024-03-18T14:41:12.925 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.923+0000 7fdb11f60640 1 -- start start 2024-03-18T14:41:12.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb11f60640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c0d0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb11f60640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0c0d0f30 con 0x7fdb0c06d1f0 2024-03-18T14:41:12.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb11f60640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0c0b1070 con 0x7fdb0c0ce930 2024-03-18T14:41:12.926 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb11f60640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0c0b1350 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb0bfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c0d0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb0b7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdb0c0ce930 0x7fdb0c06cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57410/0 (socket says 172.21.15.92:57410) 2024-03-18T14:41:12.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb0b7fe640 1 -- 172.21.15.92:0/582443374 learned_addr learned my addr 172.21.15.92:0/582443374 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-18T14:41:12.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb0affd640 1 -- 172.21.15.92:0/582443374 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3360692794 0 0) 0x7fdb0c0b1070 con 0x7fdb0c0ce930 2024-03-18T14:41:12.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb0bfff640 1 -- 172.21.15.92:0/582443374 >> v1:172.21.15.92:6789/0 conn(0x7fdb0c0ce930 legacy=0x7fdb0c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb0bfff640 1 -- 172.21.15.92:0/582443374 >> v1:172.21.15.6:6789/0 conn(0x7fdb0c06d1f0 legacy=0x7fdb0c0d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.927 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.924+0000 7fdb0bfff640 1 -- 172.21.15.92:0/582443374 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c0ced10 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.925+0000 7fdb0bfff640 1 --2- 172.21.15.92:0/582443374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c0d0850 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009dc0 tx=0x7fdb0402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa79d89fa114f8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.925+0000 7fdb0affd640 1 -- 172.21.15.92:0/582443374 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04039070 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.925+0000 7fdb0affd640 1 -- 172.21.15.92:0/582443374 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0402ccd0 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.925+0000 7fdb0affd640 1 -- 172.21.15.92:0/582443374 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04034660 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.926+0000 7fdb11f60640 1 -- 172.21.15.92:0/582443374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 msgr2=0x7fdb0c0d0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.926+0000 7fdb11f60640 1 --2- 172.21.15.92:0/582443374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c0d0850 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009dc0 tx=0x7fdb0402bf20 comp rx=0 tx=0).stop 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.926+0000 7fdb11f60640 1 -- 172.21.15.92:0/582443374 shutdown_connections 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.926+0000 7fdb11f60640 1 --2- 172.21.15.92:0/582443374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c0d0850 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.928 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.926+0000 7fdb11f60640 1 -- 172.21.15.92:0/582443374 >> 172.21.15.92:0/582443374 conn(0x7fdb0c08b0c0 msgr2=0x7fdb0c0d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:12.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.926+0000 7fdb11f60640 1 -- 172.21.15.92:0/582443374 shutdown_connections 2024-03-18T14:41:12.929 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.926+0000 7fdb11f60640 1 -- 172.21.15.92:0/582443374 wait complete. 2024-03-18T14:41:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.927+0000 7fdb11f60640 1 Processor -- start 2024-03-18T14:41:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.927+0000 7fdb11f60640 1 -- start start 2024-03-18T14:41:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb0c06d1f0 0x7fdb0c15e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c14af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb0c0ce930 0x7fdb0c14d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0c15db00 con 0x7fdb0c06d1f0 2024-03-18T14:41:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0c15d980 con 0x7fdb0c0ce930 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0c15dc80 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb0bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb0c0ce930 0x7fdb0c14d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb02ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c14af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb0bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb0c0ce930 0x7fdb0c14d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54216/0 (socket says 172.21.15.92:54216) 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb0bfff640 1 -- 172.21.15.92:0/3684177886 learned_addr learned my addr 172.21.15.92:0/3684177886 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb0b7fe640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb0c06d1f0 0x7fdb0c15e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb02ffd640 1 -- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb0c0ce930 msgr2=0x7fdb0c14d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb02ffd640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb0c0ce930 0x7fdb0c14d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb02ffd640 1 -- 172.21.15.92:0/3684177886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb0c06d1f0 msgr2=0x7fdb0c15e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb02ffd640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb0c06d1f0 0x7fdb0c15e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb02ffd640 1 -- 172.21.15.92:0/3684177886 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c14df00 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb0bfff640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb0c0ce930 0x7fdb0c14d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb02ffd640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c14af20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c3c0 tx=0x7fdaf400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb08ff9640 1 -- 172.21.15.92:0/3684177886 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4017070 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.932 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb08ff9640 1 -- 172.21.15.92:0/3684177886 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf40052d0 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb08ff9640 1 -- 172.21.15.92:0/3684177886 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4003830 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb0c15fdc0 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.928+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb0c160340 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.929+0000 7fdb08ff9640 1 -- 172.21.15.92:0/3684177886 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdaf401a070 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.929+0000 7fdb08ff9640 1 -- 172.21.15.92:0/3684177886 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fdaf4011b80 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.929+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb0c0751b0 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.933 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:12.932+0000 7fdb08ff9640 1 -- 172.21.15.92:0/3684177886 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdaf4021080 con 0x7fdb0c0b2b60 2024-03-18T14:41:12.941 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:12.939+0000 7f7d79414040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T14:41:13.006 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.004+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7390e35b-e1e4-4b0d-87da-923c0952b58c", "id": 5} v 0) -- 0x7fdb0c08b680 con 0x7fdb0c0b2b60 2024-03-18T14:41:13.011 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.009+0000 7fdb08ff9640 1 -- 172.21.15.92:0/3684177886 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7390e35b-e1e4-4b0d-87da-923c0952b58c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fdb0c14df00 con 0x7fdb0c0b2b60 2024-03-18T14:41:13.011 INFO:teuthology.orchestra.run.smithi092.stdout:5 2024-03-18T14:41:13.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 msgr2=0x7fdb0c14af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c14af20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c3c0 tx=0x7fdaf400c880 comp rx=0 tx=0).stop 2024-03-18T14:41:13.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 shutdown_connections 2024-03-18T14:41:13.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb0c0ce930 0x7fdb0c14d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb0c0b2b60 0x7fdb0c14af20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.012 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 --2- 172.21.15.92:0/3684177886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb0c06d1f0 0x7fdb0c15e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.013 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 >> 172.21.15.92:0/3684177886 conn(0x7fdb0c08b0c0 msgr2=0x7fdb0c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:13.013 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 shutdown_connections 2024-03-18T14:41:13.013 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.010+0000 7fdb11f60640 1 -- 172.21.15.92:0/3684177886 wait complete. 2024-03-18T14:41:13.014 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.012+0000 7f7d79414040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T14:41:13.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 01baa0b2-56c2-436f-893b-2282f86b7e05 6 2024-03-18T14:41:13.063 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.060+0000 7f7d79414040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T14:41:13.117 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.114+0000 7f7d79414040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T14:41:13.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.123+0000 7f40eeaa0640 1 Processor -- start 2024-03-18T14:41:13.125 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.123+0000 7f40eeaa0640 1 -- start start 2024-03-18T14:41:13.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40eeaa0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f40e80ce8d0 0x7f40e80cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40eeaa0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e8070720 con 0x7f40e80d0360 2024-03-18T14:41:13.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40eeaa0640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e8070a00 con 0x7f40e815df80 2024-03-18T14:41:13.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40eeaa0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f40e8070ce0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40e7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f40e80ce8d0 0x7f40e80cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40ec83c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f40e815df80 0x7f40e80ce1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57418/0 (socket says 172.21.15.92:57418) 2024-03-18T14:41:13.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40ec83c640 1 -- 172.21.15.92:0/1314482387 learned_addr learned my addr 172.21.15.92:0/1314482387 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-18T14:41:13.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40e7fff640 1 -- 172.21.15.92:0/1314482387 >> v1:172.21.15.92:6789/0 conn(0x7f40e815df80 legacy=0x7f40e80ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40e7fff640 1 -- 172.21.15.92:0/1314482387 >> v1:172.21.15.6:6789/0 conn(0x7f40e80d0360 legacy=0x7f40e80d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.124+0000 7f40e7fff640 1 -- 172.21.15.92:0/1314482387 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e8070fc0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.125+0000 7f40e7fff640 1 --2- 172.21.15.92:0/1314482387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e80cfe20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f40d40097a0 tx=0x7f40d402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc896bc651e28976 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:13.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.125+0000 7f40e77fe640 1 -- 172.21.15.92:0/1314482387 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d4034070 con 0x7f40e80ce8d0 2024-03-18T14:41:13.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.125+0000 7f40e77fe640 1 -- 172.21.15.92:0/1314482387 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d4030610 con 0x7f40e80ce8d0 2024-03-18T14:41:13.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.125+0000 7f40e77fe640 1 -- 172.21.15.92:0/1314482387 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d40309e0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.126+0000 7f40eeaa0640 1 -- 172.21.15.92:0/1314482387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 msgr2=0x7f40e80cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.126+0000 7f40eeaa0640 1 --2- 172.21.15.92:0/1314482387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e80cfe20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f40d40097a0 tx=0x7f40d402b960 comp rx=0 tx=0).stop 2024-03-18T14:41:13.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.126+0000 7f40eeaa0640 1 -- 172.21.15.92:0/1314482387 shutdown_connections 2024-03-18T14:41:13.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.126+0000 7f40eeaa0640 1 --2- 172.21.15.92:0/1314482387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e80cfe20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.126+0000 7f40eeaa0640 1 -- 172.21.15.92:0/1314482387 >> 172.21.15.92:0/1314482387 conn(0x7f40e80d1b50 msgr2=0x7f40e80d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:13.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.126+0000 7f40eeaa0640 1 -- 172.21.15.92:0/1314482387 shutdown_connections 2024-03-18T14:41:13.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.126+0000 7f40eeaa0640 1 -- 172.21.15.92:0/1314482387 wait complete. 2024-03-18T14:41:13.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.127+0000 7f40eeaa0640 1 Processor -- start 2024-03-18T14:41:13.129 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.127+0000 7f40eeaa0640 1 -- start start 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40eeaa0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40eeaa0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e80d0360 0x7f40e814dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40eeaa0640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40e815df80 0x7f40e81f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40eeaa0640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f40e80d29b0 con 0x7f40e80d0360 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40eeaa0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40e80d2830 con 0x7f40e80ce8d0 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40eeaa0640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f40e806d4f0 con 0x7f40e815df80 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e815ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54230/0 (socket says 172.21.15.92:54230) 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 -- 172.21.15.92:0/3754624888 learned_addr learned my addr 172.21.15.92:0/3754624888 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ed03d640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40e815df80 0x7f40e81f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40e7fff640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e80d0360 0x7f40e814dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 -- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40e815df80 msgr2=0x7f40e81f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40e815df80 0x7f40e81f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 -- 172.21.15.92:0/3754624888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e80d0360 msgr2=0x7f40e814dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e80d0360 0x7f40e814dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.128+0000 7f40ec83c640 1 -- 172.21.15.92:0/3754624888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e815f6f0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40ec83c640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e815ed70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00c480 tx=0x7f40dc00c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40e57fa640 1 -- 172.21.15.92:0/3754624888 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc017070 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40e57fa640 1 -- 172.21.15.92:0/3754624888 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40dc005ce0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40e57fa640 1 -- 172.21.15.92:0/3754624888 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0072f0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e815f9d0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e815ff90 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40e57fa640 1 -- 172.21.15.92:0/3754624888 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f40dc01a070 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.129+0000 7f40e57fa640 1 -- 172.21.15.92:0/3754624888 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f40dc01cd40 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.130+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e80751b0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.133 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.132+0000 7f40e57fa640 1 -- 172.21.15.92:0/3754624888 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f40dc0074e0 con 0x7f40e80ce8d0 2024-03-18T14:41:13.167 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.164+0000 7f7d79414040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T14:41:13.207 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.205+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "01baa0b2-56c2-436f-893b-2282f86b7e05", "id": 6} v 0) -- 0x7f40e80cff70 con 0x7f40e80ce8d0 2024-03-18T14:41:13.213 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.211+0000 7f40e57fa640 1 -- 172.21.15.92:0/3754624888 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "01baa0b2-56c2-436f-893b-2282f86b7e05", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f40dc01c440 con 0x7f40e80ce8d0 2024-03-18T14:41:13.213 INFO:teuthology.orchestra.run.smithi092.stdout:6 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 msgr2=0x7f40e815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e815ed70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00c480 tx=0x7f40dc00c940 comp rx=0 tx=0).stop 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 shutdown_connections 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40e815df80 0x7f40e81f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e80d0360 0x7f40e814dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 --2- 172.21.15.92:0/3754624888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40e80ce8d0 0x7f40e815ed70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 >> 172.21.15.92:0/3754624888 conn(0x7f40e80d1b50 msgr2=0x7f40e80d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:13.214 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 shutdown_connections 2024-03-18T14:41:13.215 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.212+0000 7f40eeaa0640 1 -- 172.21.15.92:0/3754624888 wait complete. 2024-03-18T14:41:13.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new d49cb14a-4510-423d-86a8-cb12fff9c960 7 2024-03-18T14:41:13.225 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.223+0000 7fa6eb849040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:41:13.327 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.323+0000 7fd96e214640 1 Processor -- start 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.323+0000 7fd96e214640 1 -- start start 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.323+0000 7fd96e214640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd96815dfb0 0x7fd9680b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.323+0000 7fd96e214640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9680add30 0x7fd96815e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.323+0000 7fd96e214640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9680aff50 con 0x7fd9680b0f30 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.323+0000 7fd96e214640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd9680b0560 con 0x7fd96815dfb0 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.323+0000 7fd96e214640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd9680b1f60 con 0x7fd9680add30 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd967fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9680add30 0x7fd96815e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd967fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9680add30 0x7fd96815e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:51576/0 (socket says 172.21.15.92:51576) 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd967fff640 1 -- 172.21.15.92:0/83082095 learned_addr learned my addr 172.21.15.92:0/83082095 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:13.328 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd9677fe640 1 --2- 172.21.15.92:0/83082095 >> v2:172.21.15.6:3300/0 conn(0x7fd96815dfb0 0x7fd9680b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd967fff640 1 -- 172.21.15.92:0/83082095 >> v2:172.21.15.6:3300/0 conn(0x7fd96815dfb0 msgr2=0x7fd9680b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd967fff640 1 --2- 172.21.15.92:0/83082095 >> v2:172.21.15.6:3300/0 conn(0x7fd96815dfb0 0x7fd9680b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd967fff640 1 -- 172.21.15.92:0/83082095 >> v1:172.21.15.6:6789/0 conn(0x7fd9680b0f30 legacy=0x7fd9680ad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd967fff640 1 -- 172.21.15.92:0/83082095 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9680b01f0 con 0x7fd9680add30 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.324+0000 7fd9677fe640 1 --2- 172.21.15.92:0/83082095 >> v2:172.21.15.6:3300/0 conn(0x7fd96815dfb0 0x7fd9680b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd967fff640 1 --2- 172.21.15.92:0/83082095 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9680add30 0x7fd96815e390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9680b06e0 tx=0x7fd96002bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=236fb0c265bd05b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd9667fc640 1 -- 172.21.15.92:0/83082095 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96002caf0 con 0x7fd9680add30 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd9667fc640 1 -- 172.21.15.92:0/83082095 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd96002cc90 con 0x7fd9680add30 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd9667fc640 1 -- 172.21.15.92:0/83082095 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960034610 con 0x7fd9680add30 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 -- 172.21.15.92:0/83082095 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9680add30 msgr2=0x7fd96815e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 --2- 172.21.15.92:0/83082095 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9680add30 0x7fd96815e390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9680b06e0 tx=0x7fd96002bf50 comp rx=0 tx=0).stop 2024-03-18T14:41:13.329 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 -- 172.21.15.92:0/83082095 shutdown_connections 2024-03-18T14:41:13.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 --2- 172.21.15.92:0/83082095 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9680add30 0x7fd96815e390 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 --2- 172.21.15.92:0/83082095 >> v2:172.21.15.6:3300/0 conn(0x7fd96815dfb0 0x7fd9680b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 -- 172.21.15.92:0/83082095 >> 172.21.15.92:0/83082095 conn(0x7fd9680d09b0 msgr2=0x7fd9680d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:13.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 -- 172.21.15.92:0/83082095 shutdown_connections 2024-03-18T14:41:13.330 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.325+0000 7fd96e214640 1 -- 172.21.15.92:0/83082095 wait complete. 2024-03-18T14:41:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.326+0000 7fd96e214640 1 Processor -- start 2024-03-18T14:41:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd96e214640 1 -- start start 2024-03-18T14:41:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd96e214640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9680b0f30 0x7fd96814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd96e214640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd96815dfb0 0x7fd968159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd96e214640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9681595c0 0x7fd9681f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd96e214640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd9680afc50 con 0x7fd9680b0f30 2024-03-18T14:41:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd96e214640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd9680afad0 con 0x7fd9681595c0 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd96e214640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd9680b1b10 con 0x7fd96815dfb0 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd966ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd96815dfb0 0x7fd968159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd966ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd96815dfb0 0x7fd968159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:51590/0 (socket says 172.21.15.92:51590) 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd966ffd640 1 -- 172.21.15.92:0/2222563919 learned_addr learned my addr 172.21.15.92:0/2222563919 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd967fff640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9681595c0 0x7fd9681f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.327+0000 7fd9677fe640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9680b0f30 0x7fd96814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd966ffd640 1 -- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9681595c0 msgr2=0x7fd9681f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd966ffd640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9681595c0 0x7fd9681f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd966ffd640 1 -- 172.21.15.92:0/2222563919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9680b0f30 msgr2=0x7fd96814ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd966ffd640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9680b0f30 0x7fd96814ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.334 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd966ffd640 1 -- 172.21.15.92:0/2222563919 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96815ec70 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd967fff640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9681595c0 0x7fd9681f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd966ffd640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd96815dfb0 0x7fd968159080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00c430 tx=0x7fd95c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd94ffff640 1 -- 172.21.15.92:0/2222563919 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c017070 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd94ffff640 1 -- 172.21.15.92:0/2222563919 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95c0052d0 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96815ef50 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd94ffff640 1 -- 172.21.15.92:0/2222563919 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c003850 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96815f7f0 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd94ffff640 1 -- 172.21.15.92:0/2222563919 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd95c01a070 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.328+0000 7fd94ffff640 1 -- 172.21.15.92:0/2222563919 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fd95c011b30 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.329+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd96815f9f0 con 0x7fd96815dfb0 2024-03-18T14:41:13.335 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.331+0000 7fd94ffff640 1 -- 172.21.15.92:0/2222563919 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd95c021020 con 0x7fd96815dfb0 2024-03-18T14:41:13.366 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.364+0000 7fa6eb849040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:41:13.405 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.403+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d49cb14a-4510-423d-86a8-cb12fff9c960", "id": 7} v 0) -- 0x7fd9680751b0 con 0x7fd96815dfb0 2024-03-18T14:41:13.413 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.411+0000 7fd94ffff640 1 -- 172.21.15.92:0/2222563919 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d49cb14a-4510-423d-86a8-cb12fff9c960", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fd95c011590 con 0x7fd96815dfb0 2024-03-18T14:41:13.413 INFO:teuthology.orchestra.run.smithi092.stdout:7 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd96815dfb0 msgr2=0x7fd968159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd96815dfb0 0x7fd968159080 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00c430 tx=0x7fd95c00c8f0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 shutdown_connections 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9681595c0 0x7fd9681f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd96815dfb0 0x7fd968159080 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 --2- 172.21.15.92:0/2222563919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9680b0f30 0x7fd96814ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 >> 172.21.15.92:0/2222563919 conn(0x7fd9680d09b0 msgr2=0x7fd96815e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 shutdown_connections 2024-03-18T14:41:13.415 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:41:13.412+0000 7fd96e214640 1 -- 172.21.15.92:0/2222563919 wait complete. 2024-03-18T14:41:13.425 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-18T14:41:13.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-18T14:41:13.427 INFO:tasks.ceph.osd.0:Started 2024-03-18T14:41:13.427 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-18T14:41:13.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-18T14:41:13.430 INFO:tasks.ceph.osd.1:Started 2024-03-18T14:41:13.430 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-18T14:41:13.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-18T14:41:13.432 INFO:tasks.ceph.osd.2:Started 2024-03-18T14:41:13.432 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-18T14:41:13.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-18T14:41:13.434 INFO:tasks.ceph.osd.3:Started 2024-03-18T14:41:13.434 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-18T14:41:13.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-18T14:41:13.437 INFO:tasks.ceph.osd.4:Started 2024-03-18T14:41:13.437 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-18T14:41:13.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-18T14:41:13.439 INFO:tasks.ceph.osd.5:Started 2024-03-18T14:41:13.439 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-18T14:41:13.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-18T14:41:13.441 INFO:tasks.ceph.osd.6:Started 2024-03-18T14:41:13.441 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-18T14:41:13.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-18T14:41:13.443 INFO:tasks.ceph.osd.7:Started 2024-03-18T14:41:13.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:13.486 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.485+0000 7fa6eb849040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:41:13.540 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.538+0000 7fa6eb849040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:41:13.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.575+0000 7f0d4cbc4640 1 Processor -- start 2024-03-18T14:41:13.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.575+0000 7f0d4cbc4640 1 -- start start 2024-03-18T14:41:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d4cbc4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0d480ced00 0x7f0d48162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d4cbc4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d480b9660 con 0x7f0d4815e020 2024-03-18T14:41:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d4cbc4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d480ce390 con 0x7f0d480d3210 2024-03-18T14:41:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d4cbc4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0d480d2270 con 0x7f0d480ced00 2024-03-18T14:41:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d4659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0d4815e020 0x7f0d480d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59134/0 (socket says 172.21.15.6:59134) 2024-03-18T14:41:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d4659c640 1 -- 172.21.15.6:0/3050018039 learned_addr learned my addr 172.21.15.6:0/3050018039 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d3ffff640 1 --2- 172.21.15.6:0/3050018039 >> v2:172.21.15.92:3300/0 conn(0x7f0d480ced00 0x7f0d48162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d45d9b640 1 -- 172.21.15.6:0/3050018039 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062786708 0 0) 0x7f0d480b9660 con 0x7f0d4815e020 2024-03-18T14:41:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d45d9b640 1 -- 172.21.15.6:0/3050018039 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d34003520 con 0x7f0d4815e020 2024-03-18T14:41:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d3ffff640 1 -- 172.21.15.6:0/3050018039 >> v1:172.21.15.92:6789/0 conn(0x7f0d480d3210 legacy=0x7f0d480d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d3ffff640 1 -- 172.21.15.6:0/3050018039 >> v1:172.21.15.6:6789/0 conn(0x7f0d4815e020 legacy=0x7f0d480d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.576+0000 7f0d3ffff640 1 -- 172.21.15.6:0/3050018039 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d480b9900 con 0x7f0d480ced00 2024-03-18T14:41:13.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.577+0000 7f0d3ffff640 1 --2- 172.21.15.6:0/3050018039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d48162380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d300097a0 tx=0x7f0d3002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18bdcb01c5bf0f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:13.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.577+0000 7f0d45d9b640 1 -- 172.21.15.6:0/3050018039 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30035020 con 0x7f0d480ced00 2024-03-18T14:41:13.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.577+0000 7f0d45d9b640 1 -- 172.21.15.6:0/3050018039 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d30031830 con 0x7f0d480ced00 2024-03-18T14:41:13.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.577+0000 7f0d45d9b640 1 -- 172.21.15.6:0/3050018039 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30031c00 con 0x7f0d480ced00 2024-03-18T14:41:13.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.578+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3050018039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 msgr2=0x7f0d48162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.578+0000 7f0d4cbc4640 1 --2- 172.21.15.6:0/3050018039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d48162380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d300097a0 tx=0x7f0d3002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.578+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3050018039 shutdown_connections 2024-03-18T14:41:13.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.578+0000 7f0d4cbc4640 1 --2- 172.21.15.6:0/3050018039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d48162380 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.578+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3050018039 >> 172.21.15.6:0/3050018039 conn(0x7f0d48072ce0 msgr2=0x7f0d480730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:13.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.578+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3050018039 shutdown_connections 2024-03-18T14:41:13.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.578+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3050018039 wait complete. 2024-03-18T14:41:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.579+0000 7f0d4cbc4640 1 Processor -- start 2024-03-18T14:41:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.579+0000 7f0d4cbc4640 1 -- start start 2024-03-18T14:41:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4cbc4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d481f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4cbc4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d480d3210 0x7f0d480b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4cbc4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4815e020 0x7f0d480c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4cbc4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0d480cf2c0 con 0x7f0d480d3210 2024-03-18T14:41:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4cbc4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0d480ce890 con 0x7f0d480ced00 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4cbc4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0d480cf440 con 0x7f0d4815e020 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d3ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d480d3210 0x7f0d480b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4659c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d481f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d3ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d480d3210 0x7f0d480b5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51680/0 (socket says 172.21.15.6:51680) 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d3ffff640 1 -- 172.21.15.6:0/3752942156 learned_addr learned my addr 172.21.15.6:0/3752942156 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d46d9d640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4815e020 0x7f0d480c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4659c640 1 -- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4815e020 msgr2=0x7f0d480c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4659c640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4815e020 0x7f0d480c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4659c640 1 -- 172.21.15.6:0/3752942156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d480d3210 msgr2=0x7f0d480b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.580+0000 7f0d4659c640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d480d3210 0x7f0d480b5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d4659c640 1 -- 172.21.15.6:0/3752942156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d48149cb0 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d4659c640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d481f4520 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c390 tx=0x7f0d3800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d3effd640 1 -- 172.21.15.6:0/3752942156 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38017070 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d3effd640 1 -- 172.21.15.6:0/3752942156 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d38005ce0 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d48149f90 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d3effd640 1 -- 172.21.15.6:0/3752942156 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d380072d0 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4814a550 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d3effd640 1 -- 172.21.15.6:0/3752942156 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0d3801a070 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.581+0000 7f0d3effd640 1 -- 172.21.15.6:0/3752942156 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f0d3801cc10 con 0x7f0d480ced00 2024-03-18T14:41:13.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.582+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d4815ad50 con 0x7f0d480ced00 2024-03-18T14:41:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.584+0000 7f0d3effd640 1 -- 172.21.15.6:0/3752942156 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0d380074d0 con 0x7f0d480ced00 2024-03-18T14:41:13.592 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.590+0000 7f7d79414040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T14:41:13.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.647+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d4808afe0 con 0x7f0d480ced00 2024-03-18T14:41:13.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.647+0000 7f0d3effd640 1 -- 172.21.15.6:0/3752942156 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f0d3801c440 con 0x7f0d480ced00 2024-03-18T14:41:13.650 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:13.651 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:13.405413+0000","last_up_change":"0.000000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 msgr2=0x7f0d481f4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d481f4520 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c390 tx=0x7f0d3800c850 comp rx=0 tx=0).stop 2024-03-18T14:41:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 shutdown_connections 2024-03-18T14:41:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4815e020 0x7f0d480c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d480d3210 0x7f0d480b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 --2- 172.21.15.6:0/3752942156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d480ced00 0x7f0d481f4520 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 >> 172.21.15.6:0/3752942156 conn(0x7f0d48072ce0 msgr2=0x7f0d480b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 shutdown_connections 2024-03-18T14:41:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:13.648+0000 7f0d4cbc4640 1 -- 172.21.15.6:0/3752942156 wait complete. 2024-03-18T14:41:13.652 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.650+0000 7fa6eb849040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:41:13.662 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-18T14:41:13.662 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-18T14:41:13.734 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.731+0000 7f7d79414040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T14:41:13.746 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.744+0000 7fa6eb849040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:41:13.849 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.846+0000 7f7d79414040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T14:41:13.895 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:13.893+0000 7f7d79414040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T14:41:13.900 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.898+0000 7fa6eb849040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:41:13.942 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:13.940+0000 7fa6eb849040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:41:14.012 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:14.009+0000 7f7d79414040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:41:14.040 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-18T14:41:14.037+0000 7fb5c120d640 -1 Falling back to public interface 2024-03-18T14:41:14.046 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-18T14:41:14.044+0000 7f1e11411640 -1 Falling back to public interface 2024-03-18T14:41:14.048 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-18T14:41:14.045+0000 7fae3c126640 -1 Falling back to public interface 2024-03-18T14:41:14.048 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-18T14:41:14.045+0000 7f9ecf011640 -1 Falling back to public interface 2024-03-18T14:41:14.050 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-18T14:41:14.048+0000 7f8e79011640 -1 Falling back to public interface 2024-03-18T14:41:14.052 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-18T14:41:14.049+0000 7fad197bd640 -1 Falling back to public interface 2024-03-18T14:41:14.054 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-18T14:41:14.052+0000 7f4892a11640 -1 Falling back to public interface 2024-03-18T14:41:14.060 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-18T14:41:14.058+0000 7f4879420640 -1 Falling back to public interface 2024-03-18T14:41:14.097 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:14.095+0000 7fa6eb849040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:41:14.113 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:14.110+0000 7f7d79414040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T14:41:14.274 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:14.272+0000 7f7d79414040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T14:41:14.318 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:14.315+0000 7f7d79414040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T14:41:14.468 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:14.480 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:14.477+0000 7f7d79414040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T14:41:14.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.555+0000 7f7804a44640 1 Processor -- start 2024-03-18T14:41:14.558 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.555+0000 7f7804a44640 1 -- start start 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.556+0000 7f7804a44640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f780006b850 0x7f78000b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.556+0000 7f7804a44640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78001628e0 con 0x7f780015dd10 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.556+0000 7f7804a44640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7800161b20 con 0x7f78000b1cf0 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.556+0000 7f7804a44640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7800162450 con 0x7f780006b850 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.556+0000 7f77fdd9b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f780006b850 0x7f78000b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.557+0000 7f77fe59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f780015dd10 0x7f780006b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59138/0 (socket says 172.21.15.6:59138) 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.557+0000 7f77fe59c640 1 -- 172.21.15.6:0/1381034689 learned_addr learned my addr 172.21.15.6:0/1381034689 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.557+0000 7f77fdd9b640 1 -- 172.21.15.6:0/1381034689 >> v1:172.21.15.92:6789/0 conn(0x7f78000b1cf0 legacy=0x7f78000cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.557+0000 7f77fdd9b640 1 -- 172.21.15.6:0/1381034689 >> v1:172.21.15.6:6789/0 conn(0x7f780015dd10 legacy=0x7f780006b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.557+0000 7f77fdd9b640 1 -- 172.21.15.6:0/1381034689 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7800161e00 con 0x7f780006b850 2024-03-18T14:41:14.560 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.557+0000 7f77fd59a640 1 -- 172.21.15.6:0/1381034689 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 800222463 0 0) 0x7f78001628e0 con 0x7f780015dd10 2024-03-18T14:41:14.561 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f77fdd9b640 1 --2- 172.21.15.6:0/1381034689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78000b17b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097a0 tx=0x7f77f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f057e2ca734a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f77fd59a640 1 -- 172.21.15.6:0/1381034689 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4035020 con 0x7f780006b850 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f77fd59a640 1 -- 172.21.15.6:0/1381034689 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f4031830 con 0x7f780006b850 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f77fd59a640 1 -- 172.21.15.6:0/1381034689 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4031c00 con 0x7f780006b850 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f7804a44640 1 -- 172.21.15.6:0/1381034689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 msgr2=0x7f78000b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f7804a44640 1 --2- 172.21.15.6:0/1381034689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78000b17b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097a0 tx=0x7f77f402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f7804a44640 1 -- 172.21.15.6:0/1381034689 shutdown_connections 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f7804a44640 1 --2- 172.21.15.6:0/1381034689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78000b17b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f7804a44640 1 -- 172.21.15.6:0/1381034689 >> 172.21.15.6:0/1381034689 conn(0x7f7800099a70 msgr2=0x7f7800099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f7804a44640 1 -- 172.21.15.6:0/1381034689 shutdown_connections 2024-03-18T14:41:14.562 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.558+0000 7f7804a44640 1 -- 172.21.15.6:0/1381034689 wait complete. 2024-03-18T14:41:14.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 Processor -- start 2024-03-18T14:41:14.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 -- start start 2024-03-18T14:41:14.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78001f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:14.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78000b1cf0 0x7f78000b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:14.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f780015dd10 0x7f78000b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:14.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f780015e0f0 con 0x7f780006b850 2024-03-18T14:41:14.565 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f78000b2160 con 0x7f78000b1cf0 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.560+0000 7f7804a44640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f78000b4260 con 0x7f780015dd10 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78001f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78001f46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51696/0 (socket says 172.21.15.6:51696) 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 -- 172.21.15.6:0/2068820915 learned_addr learned my addr 172.21.15.6:0/2068820915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fdd9b640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78000b1cf0 0x7f78000b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fed9d640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f780015dd10 0x7f78000b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 -- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f780015dd10 msgr2=0x7f78000b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f780015dd10 0x7f78000b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 -- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78000b1cf0 msgr2=0x7f78000b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78000b1cf0 0x7f78000b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 -- 172.21.15.6:0/2068820915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78000b4540 con 0x7f780006b850 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fdd9b640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78000b1cf0 0x7f78000b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77fe59c640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78001f46d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f77f000c4a0 tx=0x7f77f000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:14.566 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77eeffd640 1 -- 172.21.15.6:0/2068820915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0018070 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77eeffd640 1 -- 172.21.15.6:0/2068820915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f00074e0 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78000b4820 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f77eeffd640 1 -- 172.21.15.6:0/2068820915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0004a20 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.561+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78001f4c10 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.562+0000 7f77eeffd640 1 -- 172.21.15.6:0/2068820915 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f77f001b070 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.562+0000 7f77eeffd640 1 -- 172.21.15.6:0/2068820915 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f77f0012ad0 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.562+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f780009ec10 con 0x7f780006b850 2024-03-18T14:41:14.567 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.565+0000 7f77eeffd640 1 -- 172.21.15.6:0/2068820915 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f77f0004c20 con 0x7f780006b850 2024-03-18T14:41:14.630 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.627+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78000752c0 con 0x7f780006b850 2024-03-18T14:41:14.631 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.628+0000 7f77eeffd640 1 -- 172.21.15.6:0/2068820915 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f77f001d440 con 0x7f780006b850 2024-03-18T14:41:14.631 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-18T14:41:14.631 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:13.405413+0000","last_up_change":"0.000000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 msgr2=0x7f78001f46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78001f46d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f77f000c4a0 tx=0x7f77f000c960 comp rx=0 tx=0).stop 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 shutdown_connections 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f780015dd10 0x7f78000b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78000b1cf0 0x7f78000b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 --2- 172.21.15.6:0/2068820915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f780006b850 0x7f78001f46d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 >> 172.21.15.6:0/2068820915 conn(0x7f7800099a70 msgr2=0x7f7800160cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 shutdown_connections 2024-03-18T14:41:14.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:14.629+0000 7f7804a44640 1 -- 172.21.15.6:0/2068820915 wait complete. 2024-03-18T14:41:14.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-18T14:41:14.919 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:14.918+0000 7fa6eb849040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:41:15.193 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:15.191+0000 7fa6eb849040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:41:15.244 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:15.242+0000 7fa6eb849040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:41:15.340 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:15.339+0000 7f7d79414040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T14:41:15.358 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:15.356+0000 7fa6eb849040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:41:15.632 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:15.631+0000 7f7d79414040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T14:41:15.686 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:15.684+0000 7f7d79414040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T14:41:15.806 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-18T14:41:15.804+0000 7f7d79414040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T14:41:16.386 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:16.385+0000 7fa68c44c640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-18T14:41:16.387 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-18T14:41:16.385+0000 7fa68c44c640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-18T14:41:17.520 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-18T14:41:17.519+0000 7fb5c120d640 -1 osd.2 0 log_to_monitors true 2024-03-18T14:41:17.553 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-18T14:41:17.551+0000 7f1e11411640 -1 osd.6 0 log_to_monitors true 2024-03-18T14:41:17.560 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-18T14:41:17.558+0000 7f4892a11640 -1 osd.4 0 log_to_monitors true 2024-03-18T14:41:17.567 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-18T14:41:17.566+0000 7f9ecf011640 -1 osd.1 0 log_to_monitors true 2024-03-18T14:41:17.573 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-18T14:41:17.571+0000 7f8e79011640 -1 osd.5 0 log_to_monitors true 2024-03-18T14:41:17.575 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-18T14:41:17.574+0000 7f4879420640 -1 osd.7 0 log_to_monitors true 2024-03-18T14:41:17.583 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-18T14:41:17.582+0000 7fae3c126640 -1 osd.3 0 log_to_monitors true 2024-03-18T14:41:17.589 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-18T14:41:17.588+0000 7fad197bd640 -1 osd.0 0 log_to_monitors true 2024-03-18T14:41:21.396 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-18T14:41:21.394+0000 7fad15753640 -1 osd.0 0 waiting for initial osdmap 2024-03-18T14:41:21.412 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-18T14:41:21.410+0000 7fad0d573640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.428 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-18T14:41:21.426+0000 7f9ecafa7640 -1 osd.1 0 waiting for initial osdmap 2024-03-18T14:41:21.435 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-18T14:41:21.433+0000 7f4874cf8640 -1 osd.7 0 waiting for initial osdmap 2024-03-18T14:41:21.439 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-18T14:41:21.438+0000 7f9ec2dc7640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.440 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-18T14:41:21.438+0000 7fae380bc640 -1 osd.3 0 waiting for initial osdmap 2024-03-18T14:41:21.447 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:21.448 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-18T14:41:21.447+0000 7fae2fedc640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.449 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-18T14:41:21.447+0000 7f486cb18640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.473 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-18T14:41:21.471+0000 7fb5bcb3a640 -1 osd.2 0 waiting for initial osdmap 2024-03-18T14:41:21.480 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-18T14:41:21.478+0000 7f488e427640 -1 osd.4 0 waiting for initial osdmap 2024-03-18T14:41:21.484 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-18T14:41:21.481+0000 7f1e0cce5640 -1 osd.6 0 waiting for initial osdmap 2024-03-18T14:41:21.485 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-18T14:41:21.484+0000 7fb5b495a640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.498 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-18T14:41:21.496+0000 7f4886247640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.498 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-18T14:41:21.496+0000 7f1e04b05640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.520 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-18T14:41:21.518+0000 7f8e749af640 -1 osd.5 0 waiting for initial osdmap 2024-03-18T14:41:21.537 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-18T14:41:21.536+0000 7f8e6c7cf640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T14:41:21.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.537+0000 7f002c377640 1 Processor -- start 2024-03-18T14:41:21.539 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.537+0000 7f002c377640 1 -- start start 2024-03-18T14:41:21.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002c377640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00240cf210 0x7f0024161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:21.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002c377640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f00240b96c0 0x7f00240d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:21.540 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002c377640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00240b6ed0 con 0x7f002415e200 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002c377640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f00240b6690 con 0x7f00240b96c0 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002c377640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f00241622a0 con 0x7f00240cf210 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002a914640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f00240b96c0 0x7f00240d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002a914640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f00240b96c0 0x7f00240d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51742/0 (socket says 172.21.15.6:51742) 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002a914640 1 -- 172.21.15.6:0/2105206431 learned_addr learned my addr 172.21.15.6:0/2105206431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f0029912640 1 --2- 172.21.15.6:0/2105206431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00240cf210 0x7f0024161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002a914640 1 -- 172.21.15.6:0/2105206431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00240cf210 msgr2=0x7f0024161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002a914640 1 --2- 172.21.15.6:0/2105206431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00240cf210 0x7f0024161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002a914640 1 -- 172.21.15.6:0/2105206431 >> v1:172.21.15.92:6789/0 conn(0x7f002415e200 legacy=0x7f00240ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:21.541 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.538+0000 7f002a914640 1 -- 172.21.15.6:0/2105206431 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0024162580 con 0x7f00240b96c0 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.539+0000 7f002a914640 1 --2- 172.21.15.6:0/2105206431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240d2830 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f001c009870 tx=0x7f001c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85e93d81100fe55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.539+0000 7f0029111640 1 -- 172.21.15.6:0/2105206431 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c035020 con 0x7f00240b96c0 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.539+0000 7f0029111640 1 -- 172.21.15.6:0/2105206431 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001c031640 con 0x7f00240b96c0 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.539+0000 7f0029111640 1 -- 172.21.15.6:0/2105206431 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c02d040 con 0x7f00240b96c0 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 -- 172.21.15.6:0/2105206431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 msgr2=0x7f00240d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 --2- 172.21.15.6:0/2105206431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240d2830 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f001c009870 tx=0x7f001c02b8f0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 -- 172.21.15.6:0/2105206431 shutdown_connections 2024-03-18T14:41:21.543 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 --2- 172.21.15.6:0/2105206431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240d2830 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 --2- 172.21.15.6:0/2105206431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00240cf210 0x7f0024161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 -- 172.21.15.6:0/2105206431 >> 172.21.15.6:0/2105206431 conn(0x7f00240d0bd0 msgr2=0x7f00240d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:21.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 -- 172.21.15.6:0/2105206431 shutdown_connections 2024-03-18T14:41:21.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.540+0000 7f002c377640 1 -- 172.21.15.6:0/2105206431 wait complete. 2024-03-18T14:41:21.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.541+0000 7f002c377640 1 Processor -- start 2024-03-18T14:41:21.544 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.541+0000 7f002c377640 1 -- start start 2024-03-18T14:41:21.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:21.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00240cf210 0x7f002415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:21.546 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f002415e200 0x7f0024150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f00240d2d70 con 0x7f00240b96c0 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f00240d2310 con 0x7f00240cf210 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f00240d2ef0 con 0x7f002415e200 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51748/0 (socket says 172.21.15.6:51748) 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 -- 172.21.15.6:0/2777928447 learned_addr learned my addr 172.21.15.6:0/2777928447 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a914640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f002415e200 0x7f0024150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f0029912640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00240cf210 0x7f002415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 -- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f002415e200 msgr2=0x7f0024150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f002415e200 0x7f0024150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 -- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00240cf210 msgr2=0x7f002415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00240cf210 0x7f002415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 -- 172.21.15.6:0/2777928447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0024151f80 con 0x7f00240b96c0 2024-03-18T14:41:21.547 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002a113640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240c8270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f002000c730 tx=0x7f002000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f0016ffd640 1 -- 172.21.15.6:0/2777928447 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020016020 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f0016ffd640 1 -- 172.21.15.6:0/2777928447 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0020006d50 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0024152260 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f0016ffd640 1 -- 172.21.15.6:0/2777928447 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020003bb0 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.542+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00240c87b0 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.543+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f002408b000 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.544+0000 7f0016ffd640 1 -- 172.21.15.6:0/2777928447 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f002000a850 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.544+0000 7f0016ffd640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000806bb40 0x7f000806e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.544+0000 7f0016ffd640 1 -- 172.21.15.6:0/2777928447 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f0020089620 con 0x7f00240b96c0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.544+0000 7f0029912640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000806bb40 0x7f000806e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:21.548 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.545+0000 7f0029912640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000806bb40 0x7f000806e000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f002415ccf0 tx=0x7f0018009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:21.550 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.549+0000 7f0016ffd640 1 -- 172.21.15.6:0/2777928447 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0020020020 con 0x7f00240b96c0 2024-03-18T14:41:21.731 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.730+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00240752c0 con 0x7f00240b96c0 2024-03-18T14:41:21.732 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.730+0000 7f0016ffd640 1 -- 172.21.15.6:0/2777928447 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f00240b6ed0 con 0x7f00240b96c0 2024-03-18T14:41:21.732 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-18T14:41:21.732 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":12,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:19.387099+0000","last_up_change":"0.000000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:21.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000806bb40 msgr2=0x7f000806e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:21.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000806bb40 0x7f000806e000 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f002415ccf0 tx=0x7f0018009190 comp rx=0 tx=0).stop 2024-03-18T14:41:21.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 msgr2=0x7f00240c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:21.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240c8270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f002000c730 tx=0x7f002000cbf0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 shutdown_connections 2024-03-18T14:41:21.734 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000806bb40 0x7f000806e000 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f002415e200 0x7f0024150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00240cf210 0x7f002415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 --2- 172.21.15.6:0/2777928447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00240b96c0 0x7f00240c8270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:21.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 >> 172.21.15.6:0/2777928447 conn(0x7f00240d0bd0 msgr2=0x7f00240b5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:21.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 shutdown_connections 2024-03-18T14:41:21.735 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:21.732+0000 7f002c377640 1 -- 172.21.15.6:0/2777928447 wait complete. 2024-03-18T14:41:21.743 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-18T14:41:28.550 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:28.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.649+0000 7fb04b36f640 1 Processor -- start 2024-03-18T14:41:28.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.649+0000 7fb04b36f640 1 -- start start 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.650+0000 7fb04b36f640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.650+0000 7fb04b36f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb04415e4a0 con 0x7fb0440d0ad0 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.650+0000 7fb04b36f640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0440b59f0 con 0x7fb04415e0c0 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.650+0000 7fb04b36f640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb0440d0f40 con 0x7fb0440b6ee0 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.650+0000 7fb04990c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb0440d0ad0 0x7fb044161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57476/0 (socket says 172.21.15.6:57476) 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.650+0000 7fb04990c640 1 -- 172.21.15.6:0/3064621112 learned_addr learned my addr 172.21.15.6:0/3064621112 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb04890a640 1 --2- 172.21.15.6:0/3064621112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223390904 0 0) 0x7fb04415e4a0 con 0x7fb0440d0ad0 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb02c003520 con 0x7fb0440d0ad0 2024-03-18T14:41:28.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 684356486 0 0) 0x7fb0440b59f0 con 0x7fb04415e0c0 2024-03-18T14:41:28.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb04415e4a0 con 0x7fb04415e0c0 2024-03-18T14:41:28.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb04890a640 1 -- 172.21.15.6:0/3064621112 >> v1:172.21.15.92:6789/0 conn(0x7fb04415e0c0 legacy=0x7fb0440ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb04890a640 1 -- 172.21.15.6:0/3064621112 >> v1:172.21.15.6:6789/0 conn(0x7fb0440d0ad0 legacy=0x7fb044161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3106361047 0 0) 0x7fb02c003520 con 0x7fb0440d0ad0 2024-03-18T14:41:28.654 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.651+0000 7fb04890a640 1 -- 172.21.15.6:0/3064621112 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0440b5cd0 con 0x7fb0440b6ee0 2024-03-18T14:41:28.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.652+0000 7fb04890a640 1 --2- 172.21.15.6:0/3064621112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440d0590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb0380097d0 tx=0x7fb03802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc534a200f05e3db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:28.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.652+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0340019f0 con 0x7fb0440b6ee0 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.652+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb038037070 con 0x7fb0440b6ee0 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.652+0000 7fb043fff640 1 -- 172.21.15.6:0/3064621112 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038031780 con 0x7fb0440b6ee0 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.653+0000 7fb04b36f640 1 -- 172.21.15.6:0/3064621112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 msgr2=0x7fb0440d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.653+0000 7fb04b36f640 1 --2- 172.21.15.6:0/3064621112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440d0590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb0380097d0 tx=0x7fb03802b8a0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.653+0000 7fb04b36f640 1 -- 172.21.15.6:0/3064621112 shutdown_connections 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.653+0000 7fb04b36f640 1 --2- 172.21.15.6:0/3064621112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440d0590 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.653+0000 7fb04b36f640 1 -- 172.21.15.6:0/3064621112 >> 172.21.15.6:0/3064621112 conn(0x7fb04406c9d0 msgr2=0x7fb04406cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.653+0000 7fb04b36f640 1 -- 172.21.15.6:0/3064621112 shutdown_connections 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.653+0000 7fb04b36f640 1 -- 172.21.15.6:0/3064621112 wait complete. 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.654+0000 7fb04b36f640 1 Processor -- start 2024-03-18T14:41:28.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.654+0000 7fb04b36f640 1 -- start start 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04b36f640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04b36f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0440d0ad0 0x7fb0441532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04b36f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb04415e0c0 0x7fb0440c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04b36f640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb0440b6290 con 0x7fb0440d0ad0 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04b36f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb0440b57a0 con 0x7fb04415e0c0 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04b36f640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb0440b6570 con 0x7fb0440b6ee0 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0440d0ad0 0x7fb0441532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0440d0ad0 0x7fb0441532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38224/0 (socket says 172.21.15.6:38224) 2024-03-18T14:41:28.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 -- 172.21.15.6:0/2539201398 learned_addr learned my addr 172.21.15.6:0/2539201398 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04910b640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04990c640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb04415e0c0 0x7fb0440c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 -- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 msgr2=0x7fb0440c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 -- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb04415e0c0 msgr2=0x7fb0440c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb04415e0c0 0x7fb0440c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 -- 172.21.15.6:0/2539201398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0441537f0 con 0x7fb0440d0ad0 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb04890a640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0440d0ad0 0x7fb0441532b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb03802b8e0 tx=0x7fb0380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.655+0000 7fb041ffb640 1 -- 172.21.15.6:0/2539201398 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038032610 con 0x7fb0440d0ad0 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.656+0000 7fb041ffb640 1 -- 172.21.15.6:0/2539201398 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0380327b0 con 0x7fb0440d0ad0 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.656+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0440c8060 con 0x7fb0440d0ad0 2024-03-18T14:41:28.660 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.656+0000 7fb041ffb640 1 -- 172.21.15.6:0/2539201398 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038032ba0 con 0x7fb0440d0ad0 2024-03-18T14:41:28.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.656+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0440c8620 con 0x7fb0440d0ad0 2024-03-18T14:41:28.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.657+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04408b000 con 0x7fb0440d0ad0 2024-03-18T14:41:28.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.657+0000 7fb041ffb640 1 -- 172.21.15.6:0/2539201398 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb03804c060 con 0x7fb0440d0ad0 2024-03-18T14:41:28.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.658+0000 7fb041ffb640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb03006bc00 0x7fb03006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.658+0000 7fb041ffb640 1 -- 172.21.15.6:0/2539201398 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fb0380abd60 con 0x7fb0440d0ad0 2024-03-18T14:41:28.662 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.658+0000 7fb04910b640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb03006bc00 0x7fb03006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.662 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.659+0000 7fb04910b640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb03006bc00 0x7fb03006e0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0011e0 tx=0x7fb03c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:28.664 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.663+0000 7fb041ffb640 1 -- 172.21.15.6:0/2539201398 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb0441537f0 con 0x7fb0440d0ad0 2024-03-18T14:41:28.845 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.843+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0440752a0 con 0x7fb0440d0ad0 2024-03-18T14:41:28.846 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.844+0000 7fb041ffb640 1 -- 172.21.15.6:0/2539201398 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12788 (secure 0 0 0) 0x7fb038040070 con 0x7fb0440d0ad0 2024-03-18T14:41:28.846 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-18T14:41:28.846 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":17,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:26.411799+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb03006bc00 msgr2=0x7fb03006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb03006bc00 0x7fb03006e0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0011e0 tx=0x7fb03c009040 comp rx=0 tx=0).stop 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0440d0ad0 msgr2=0x7fb0441532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0440d0ad0 0x7fb0441532b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb03802b8e0 tx=0x7fb0380046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 shutdown_connections 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb03006bc00 0x7fb03006e0c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb04415e0c0 0x7fb0440c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0440d0ad0 0x7fb0441532b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.848 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 --2- 172.21.15.6:0/2539201398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0440b6ee0 0x7fb0440c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 >> 172.21.15.6:0/2539201398 conn(0x7fb04406c9d0 msgr2=0x7fb0440cee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:28.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 shutdown_connections 2024-03-18T14:41:28.849 INFO:teuthology.misc.health.smithi006.stderr:2024-03-18T14:41:28.846+0000 7fb04b36f640 1 -- 172.21.15.6:0/2539201398 wait complete. 2024-03-18T14:41:28.857 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-18T14:41:28.857 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-18T14:41:28.858 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T14:41:28.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T14:41:28.860 INFO:tasks.ceph.mds.a:Started 2024-03-18T14:41:28.860 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T14:41:28.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T14:41:28.862 INFO:tasks.ceph.mds.c:Started 2024-03-18T14:41:28.862 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T14:41:28.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T14:41:28.865 INFO:tasks.ceph.mds.b:Started 2024-03-18T14:41:28.865 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T14:41:28.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T14:41:28.867 INFO:tasks.ceph.mds.d:Started 2024-03-18T14:41:28.868 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-18T14:41:28.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:28.936 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-18T14:41:28.936 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-18T14:41:28.953 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2024-03-18T14:41:28.953 INFO:tasks.ceph.mds.d.smithi092.stdout:starting mds.d at 2024-03-18T14:41:28.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.973+0000 7fb7c7297640 1 Processor -- start 2024-03-18T14:41:28.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c7297640 1 -- start start 2024-03-18T14:41:28.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c7297640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb7c00ade40 0x7fb7c00cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c7297640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7c00ce260 0x7fb7c0072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c7297640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7c015e430 con 0x7fb7c015e050 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c7297640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb7c00ae2d0 con 0x7fb7c00ce260 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c5033640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb7c015e050 0x7fb7c006f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57482/0 (socket says 172.21.15.6:57482) 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c7297640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb7c00ad080 con 0x7fb7c00ade40 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c5834640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7c00ce260 0x7fb7c0072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.974+0000 7fb7c5033640 1 -- 172.21.15.6:0/540746820 learned_addr learned my addr 172.21.15.6:0/540746820 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.975+0000 7fb7b7fff640 1 -- 172.21.15.6:0/540746820 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904548267 0 0) 0x7fb7c015e430 con 0x7fb7c015e050 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.975+0000 7fb7c4832640 1 --2- 172.21.15.6:0/540746820 >> v2:172.21.15.92:3300/0 conn(0x7fb7c00ade40 0x7fb7c00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.975+0000 7fb7b7fff640 1 -- 172.21.15.6:0/540746820 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7ac003520 con 0x7fb7c015e050 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.975+0000 7fb7c5834640 1 -- 172.21.15.6:0/540746820 >> v2:172.21.15.92:3300/0 conn(0x7fb7c00ade40 msgr2=0x7fb7c00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:28.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.975+0000 7fb7c5834640 1 --2- 172.21.15.6:0/540746820 >> v2:172.21.15.92:3300/0 conn(0x7fb7c00ade40 0x7fb7c00cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.975+0000 7fb7c5834640 1 -- 172.21.15.6:0/540746820 >> v1:172.21.15.6:6789/0 conn(0x7fb7c015e050 legacy=0x7fb7c006f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.975+0000 7fb7c5834640 1 -- 172.21.15.6:0/540746820 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c00ad360 con 0x7fb7c00ce260 2024-03-18T14:41:28.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c5834640 1 --2- 172.21.15.6:0/540746820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ce260 0x7fb7c0072220 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00ae7b0 tx=0x7fb7b802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a887dfccbecf10ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:28.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7b7fff640 1 -- 172.21.15.6:0/540746820 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8039070 con 0x7fb7c00ce260 2024-03-18T14:41:28.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7b7fff640 1 -- 172.21.15.6:0/540746820 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b802cc70 con 0x7fb7c00ce260 2024-03-18T14:41:28.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7b7fff640 1 -- 172.21.15.6:0/540746820 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8034640 con 0x7fb7c00ce260 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 -- 172.21.15.6:0/540746820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ce260 msgr2=0x7fb7c0072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 --2- 172.21.15.6:0/540746820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ce260 0x7fb7c0072220 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00ae7b0 tx=0x7fb7b802bec0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 -- 172.21.15.6:0/540746820 shutdown_connections 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 --2- 172.21.15.6:0/540746820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ce260 0x7fb7c0072220 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 --2- 172.21.15.6:0/540746820 >> v2:172.21.15.92:3300/0 conn(0x7fb7c00ade40 0x7fb7c00cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 -- 172.21.15.6:0/540746820 >> 172.21.15.6:0/540746820 conn(0x7fb7c00d0150 msgr2=0x7fb7c00d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 -- 172.21.15.6:0/540746820 shutdown_connections 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.976+0000 7fb7c7297640 1 -- 172.21.15.6:0/540746820 wait complete. 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.977+0000 7fb7c7297640 1 Processor -- start 2024-03-18T14:41:28.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.977+0000 7fb7c7297640 1 -- start start 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c7297640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ade40 0x7fb7c009c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c7297640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c00ce260 0x7fb7c01faf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c7297640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7c015e050 0x7fb7c01fb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c7297640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c00ce690 con 0x7fb7c00ade40 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c7297640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c00ad920 con 0x7fb7c00ce260 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c7297640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c00ce810 con 0x7fb7c015e050 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c5033640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ade40 0x7fb7c009c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c5033640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ade40 0x7fb7c009c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38262/0 (socket says 172.21.15.6:38262) 2024-03-18T14:41:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c5033640 1 -- 172.21.15.6:0/706888162 learned_addr learned my addr 172.21.15.6:0/706888162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c4832640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c00ce260 0x7fb7c01faf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.978+0000 7fb7c5834640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7c015e050 0x7fb7c01fb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c5033640 1 -- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7c015e050 msgr2=0x7fb7c01fb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c5033640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7c015e050 0x7fb7c01fb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c5033640 1 -- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c00ce260 msgr2=0x7fb7c01faf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c5033640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c00ce260 0x7fb7c01faf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c5033640 1 -- 172.21.15.6:0/706888162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c009b6c0 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c5033640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ade40 0x7fb7c009c760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c500 tx=0x7fb7bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7b5ffb640 1 -- 172.21.15.6:0/706888162 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc017070 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7b5ffb640 1 -- 172.21.15.6:0/706888162 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7bc005ce0 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7b5ffb640 1 -- 172.21.15.6:0/706888162 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc007330 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c009b9a0 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.979+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c009bf60 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.980+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c009e260 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.981+0000 7fb7b5ffb640 1 -- 172.21.15.6:0/706888162 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb7bc01a070 con 0x7fb7c00ade40 2024-03-18T14:41:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.981+0000 7fb7b5ffb640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb7a406bc00 0x7fb7a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:28.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.981+0000 7fb7b5ffb640 1 -- 172.21.15.6:0/706888162 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fb7bc089ae0 con 0x7fb7c00ade40 2024-03-18T14:41:28.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.981+0000 7fb7c4832640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb7a406bc00 0x7fb7a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:28.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.982+0000 7fb7c4832640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb7a406bc00 0x7fb7a406e0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00ccba0 tx=0x7fb7b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:28.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:28.985+0000 7fb7b5ffb640 1 -- 172.21.15.6:0/706888162 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb7bc058420 con 0x7fb7c00ade40 2024-03-18T14:41:29.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.169+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7c004f620 con 0x7fb7c00ade40 2024-03-18T14:41:29.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.170+0000 7fb7b5ffb640 1 -- 172.21.15.6:0/706888162 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12788 (secure 0 0 0) 0x7fb7bc05c4c0 con 0x7fb7c00ade40 2024-03-18T14:41:29.172 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:29.172 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:26.411799+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:29.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb7a406bc00 msgr2=0x7fb7a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb7a406bc00 0x7fb7a406e0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00ccba0 tx=0x7fb7b0008040 comp rx=0 tx=0).stop 2024-03-18T14:41:29.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ade40 msgr2=0x7fb7c009c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ade40 0x7fb7c009c760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c500 tx=0x7fb7bc00c9c0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 shutdown_connections 2024-03-18T14:41:29.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb7a406bc00 0x7fb7a406e0c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7c015e050 0x7fb7c01fb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c00ce260 0x7fb7c01faf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 --2- 172.21.15.6:0/706888162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c00ade40 0x7fb7c009c760 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 >> 172.21.15.6:0/706888162 conn(0x7fb7c00d0150 msgr2=0x7fb7c00b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:29.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 shutdown_connections 2024-03-18T14:41:29.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.172+0000 7fb7c7297640 1 -- 172.21.15.6:0/706888162 wait complete. 2024-03-18T14:41:29.184 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:41:29.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:41:29.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.287+0000 7f2215759640 1 Processor -- start 2024-03-18T14:41:29.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.287+0000 7f2215759640 1 -- start start 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f2215759640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f221015dfd0 0x7f22100b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f2215759640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22100b7e00 0x7f22100b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f2215759640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f221015e3b0 con 0x7f22100b5740 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f2215759640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f22100b8240 con 0x7f22100b7e00 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f2215759640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f22100b8470 con 0x7f221015dfd0 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22100b7e00 0x7f22100b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22100b7e00 0x7f22100b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38264/0 (socket says 172.21.15.6:38264) 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220f7fe640 1 -- 172.21.15.6:0/2077098484 learned_addr learned my addr 172.21.15.6:0/2077098484 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:29.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220effd640 1 --2- 172.21.15.6:0/2077098484 >> v2:172.21.15.92:3300/0 conn(0x7f221015dfd0 0x7f22100b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220f7fe640 1 -- 172.21.15.6:0/2077098484 >> v2:172.21.15.92:3300/0 conn(0x7f221015dfd0 msgr2=0x7f22100b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220f7fe640 1 --2- 172.21.15.6:0/2077098484 >> v2:172.21.15.92:3300/0 conn(0x7f221015dfd0 0x7f22100b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220f7fe640 1 -- 172.21.15.6:0/2077098484 >> v1:172.21.15.92:6789/0 conn(0x7f22100b5740 legacy=0x7f22100d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220f7fe640 1 -- 172.21.15.6:0/2077098484 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22100b8750 con 0x7f22100b7e00 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.288+0000 7f220effd640 1 --2- 172.21.15.6:0/2077098484 >> v2:172.21.15.92:3300/0 conn(0x7f221015dfd0 0x7f22100b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.289+0000 7f220f7fe640 1 --2- 172.21.15.6:0/2077098484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100b5bd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f22100d17c0 tx=0x7f220802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2365971d3a5313 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.289+0000 7f220e7fc640 1 -- 172.21.15.6:0/2077098484 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220802ca80 con 0x7f22100b7e00 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.289+0000 7f220e7fc640 1 -- 172.21.15.6:0/2077098484 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220802cc20 con 0x7f22100b7e00 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.289+0000 7f220e7fc640 1 -- 172.21.15.6:0/2077098484 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22080343f0 con 0x7f22100b7e00 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 -- 172.21.15.6:0/2077098484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 msgr2=0x7f22100b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 --2- 172.21.15.6:0/2077098484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100b5bd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f22100d17c0 tx=0x7f220802bee0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 -- 172.21.15.6:0/2077098484 shutdown_connections 2024-03-18T14:41:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 --2- 172.21.15.6:0/2077098484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100b5bd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 --2- 172.21.15.6:0/2077098484 >> v2:172.21.15.92:3300/0 conn(0x7f221015dfd0 0x7f22100b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 -- 172.21.15.6:0/2077098484 >> 172.21.15.6:0/2077098484 conn(0x7f2210072d10 msgr2=0x7f2210073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:29.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 -- 172.21.15.6:0/2077098484 shutdown_connections 2024-03-18T14:41:29.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.290+0000 7f2215759640 1 -- 172.21.15.6:0/2077098484 wait complete. 2024-03-18T14:41:29.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 Processor -- start 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 -- start start 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22100b5740 0x7f22100c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f221015dfd0 0x7f22101520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f22100b9b90 con 0x7f22100b7e00 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22100b8d10 con 0x7f221015dfd0 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.291+0000 7f2215759640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f22100d2550 con 0x7f22100b5740 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38280/0 (socket says 172.21.15.6:38280) 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 -- 172.21.15.6:0/4175562514 learned_addr learned my addr 172.21.15.6:0/4175562514 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f220f7fe640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f221015dfd0 0x7f22101520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f220effd640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22100b5740 0x7f22100c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 -- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22100b5740 msgr2=0x7f22100c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22100b5740 0x7f22100c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 -- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f221015dfd0 msgr2=0x7f22101520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f221015dfd0 0x7f22101520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 -- 172.21.15.6:0/4175562514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22100c8120 con 0x7f22100b7e00 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2207fff640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100cbf60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c430 tx=0x7f21f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f22077fe640 1 -- 172.21.15.6:0/4175562514 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8011790 con 0x7f22100b7e00 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f22077fe640 1 -- 172.21.15.6:0/4175562514 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f8017070 con 0x7f22100b7e00 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f22077fe640 1 -- 172.21.15.6:0/4175562514 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8011bb0 con 0x7f22100b7e00 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22100c8400 con 0x7f22100b7e00 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.292+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22100c8ce0 con 0x7f22100b7e00 2024-03-18T14:41:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.293+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22100751b0 con 0x7f22100b7e00 2024-03-18T14:41:29.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.293+0000 7f22077fe640 1 -- 172.21.15.6:0/4175562514 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f21f801c460 con 0x7f22100b7e00 2024-03-18T14:41:29.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.294+0000 7f22077fe640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f21f006bbd0 0x7f21f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.294+0000 7f22077fe640 1 -- 172.21.15.6:0/4175562514 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f21f808a5c0 con 0x7f22100b7e00 2024-03-18T14:41:29.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.294+0000 7f220effd640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f21f006bbd0 0x7f21f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.294+0000 7f220effd640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f21f006bbd0 0x7f21f006e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f22101f6d10 tx=0x7f2200008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:29.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.298+0000 7f22077fe640 1 -- 172.21.15.6:0/4175562514 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f21f8059400 con 0x7f22100b7e00 2024-03-18T14:41:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.479+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f22100d19f0 con 0x7f22100b7e00 2024-03-18T14:41:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.480+0000 7f22077fe640 1 -- 172.21.15.6:0/4175562514 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f21f8058d00 con 0x7f22100b7e00 2024-03-18T14:41:29.482 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.481+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f21f006bbd0 msgr2=0x7f21f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.481+0000 7f2215759640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f21f006bbd0 0x7f21f006e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f22101f6d10 tx=0x7f2200008040 comp rx=0 tx=0).stop 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.481+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 msgr2=0x7f22100cbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.481+0000 7f2215759640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100cbf60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c430 tx=0x7f21f800c8f0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 shutdown_connections 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f21f006bbd0 0x7f21f006e090 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f221015dfd0 0x7f22101520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22100b7e00 0x7f22100cbf60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 --2- 172.21.15.6:0/4175562514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22100b5740 0x7f22100c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 >> 172.21.15.6:0/4175562514 conn(0x7f2210072d10 msgr2=0x7f22100b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:29.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 shutdown_connections 2024-03-18T14:41:29.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.482+0000 7f2215759640 1 -- 172.21.15.6:0/4175562514 wait complete. 2024-03-18T14:41:29.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:41:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.593+0000 7fe21ca7e640 1 Processor -- start 2024-03-18T14:41:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.593+0000 7fe21ca7e640 1 -- start start 2024-03-18T14:41:29.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe21ca7e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe21815e2d0 0x7fe2180b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe21ca7e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe21815e6b0 0x7fe2180b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe21ca7e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2180b3d40 con 0x7fe2180b1f40 2024-03-18T14:41:29.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe21ca7e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe2180cd9f0 con 0x7fe21815e6b0 2024-03-18T14:41:29.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe21ca7e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe2180cdc40 con 0x7fe21815e2d0 2024-03-18T14:41:29.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe20ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe21815e6b0 0x7fe2180b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe20ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe21815e6b0 0x7fe2180b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38292/0 (socket says 172.21.15.6:38292) 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe216d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2180b1f40 0x7fe2180cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57492/0 (socket says 172.21.15.6:57492) 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.594+0000 7fe20ffff640 1 -- 172.21.15.6:0/891965290 learned_addr learned my addr 172.21.15.6:0/891965290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe21659c640 1 --2- 172.21.15.6:0/891965290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe21815e2d0 0x7fe2180b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe20ffff640 1 -- 172.21.15.6:0/891965290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe21815e2d0 msgr2=0x7fe2180b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe20ffff640 1 --2- 172.21.15.6:0/891965290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe21815e2d0 0x7fe2180b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe20ffff640 1 -- 172.21.15.6:0/891965290 >> v1:172.21.15.6:6789/0 conn(0x7fe2180b1f40 legacy=0x7fe2180cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe215d9b640 1 -- 172.21.15.6:0/891965290 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2071333354 0 0) 0x7fe2180b3d40 con 0x7fe2180b1f40 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe20ffff640 1 -- 172.21.15.6:0/891965290 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2180cdf20 con 0x7fe21815e6b0 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe21659c640 1 --2- 172.21.15.6:0/891965290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe21815e2d0 0x7fe2180b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.595+0000 7fe20ffff640 1 --2- 172.21.15.6:0/891965290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe21815e6b0 0x7fe2180b1a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe200009870 tx=0x7fe20002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863f7fd94ccb209 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe215d9b640 1 -- 172.21.15.6:0/891965290 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200034070 con 0x7fe21815e6b0 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe215d9b640 1 -- 172.21.15.6:0/891965290 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe200030840 con 0x7fe21815e6b0 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe215d9b640 1 -- 172.21.15.6:0/891965290 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200030c10 con 0x7fe21815e6b0 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 -- 172.21.15.6:0/891965290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe21815e6b0 msgr2=0x7fe2180b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/891965290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe21815e6b0 0x7fe2180b1a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe200009870 tx=0x7fe20002b980 comp rx=0 tx=0).stop 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 -- 172.21.15.6:0/891965290 shutdown_connections 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/891965290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe21815e6b0 0x7fe2180b1a00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/891965290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe21815e2d0 0x7fe2180b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 -- 172.21.15.6:0/891965290 >> 172.21.15.6:0/891965290 conn(0x7fe2180d09b0 msgr2=0x7fe2180d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 -- 172.21.15.6:0/891965290 shutdown_connections 2024-03-18T14:41:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.596+0000 7fe21ca7e640 1 -- 172.21.15.6:0/891965290 wait complete. 2024-03-18T14:41:29.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 Processor -- start 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 -- start start 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2180b1f40 0x7fe2180c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe21815e2d0 0x7fe2180c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe218145b70 0x7fe218146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe2180d1790 con 0x7fe218145b70 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe2180cb9a0 con 0x7fe21815e2d0 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe21ca7e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe2180cb5c0 con 0x7fe2180b1f40 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe216d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe218145b70 0x7fe218146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe216d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe218145b70 0x7fe218146ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38308/0 (socket says 172.21.15.6:38308) 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.598+0000 7fe216d9d640 1 -- 172.21.15.6:0/2250560711 learned_addr learned my addr 172.21.15.6:0/2250560711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe21659c640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2180b1f40 0x7fe2180c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe20ffff640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe21815e2d0 0x7fe2180c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe216d9d640 1 -- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2180b1f40 msgr2=0x7fe2180c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe216d9d640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2180b1f40 0x7fe2180c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe216d9d640 1 -- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe21815e2d0 msgr2=0x7fe2180c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe216d9d640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe21815e2d0 0x7fe2180c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe216d9d640 1 -- 172.21.15.6:0/2250560711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2180c89b0 con 0x7fe218145b70 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe20ffff640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe21815e2d0 0x7fe2180c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe216d9d640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe218145b70 0x7fe218146ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe21000c730 tx=0x7fe21000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe20effd640 1 -- 172.21.15.6:0/2250560711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210018070 con 0x7fe218145b70 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe20effd640 1 -- 172.21.15.6:0/2250560711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe21000ce40 con 0x7fe218145b70 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe20effd640 1 -- 172.21.15.6:0/2250560711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21000a860 con 0x7fe218145b70 2024-03-18T14:41:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2180c8060 con 0x7fe218145b70 2024-03-18T14:41:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.599+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2181466d0 con 0x7fe218145b70 2024-03-18T14:41:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.600+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2180681a0 con 0x7fe218145b70 2024-03-18T14:41:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.600+0000 7fe20effd640 1 -- 172.21.15.6:0/2250560711 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe21001b070 con 0x7fe218145b70 2024-03-18T14:41:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.601+0000 7fe20effd640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe1f806bc00 0x7fe1f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.601+0000 7fe20effd640 1 -- 172.21.15.6:0/2250560711 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fe210089bd0 con 0x7fe218145b70 2024-03-18T14:41:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.601+0000 7fe21659c640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe1f806bc00 0x7fe1f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.601+0000 7fe21659c640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe1f806bc00 0x7fe1f806e0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe21806ed60 tx=0x7fe208008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:29.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.605+0000 7fe20effd640 1 -- 172.21.15.6:0/2250560711 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe2180681a0 con 0x7fe218145b70 2024-03-18T14:41:29.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.826+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe21809e9a0 con 0x7fe218145b70 2024-03-18T14:41:29.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.827+0000 7fe20effd640 1 -- 172.21.15.6:0/2250560711 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1386 (secure 0 0 0) 0x7fe210058aa0 con 0x7fe218145b70 2024-03-18T14:41:29.829 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:29.829 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T14:41:29.829 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 2 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.828+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe1f806bc00 msgr2=0x7fe1f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.828+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe1f806bc00 0x7fe1f806e0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe21806ed60 tx=0x7fe208008040 comp rx=0 tx=0).stop 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.828+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe218145b70 msgr2=0x7fe218146ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.828+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe218145b70 0x7fe218146ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe21000c730 tx=0x7fe21000cbf0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 shutdown_connections 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe1f806bc00 0x7fe1f806e0c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe218145b70 0x7fe218146ed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe21815e2d0 0x7fe2180c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 --2- 172.21.15.6:0/2250560711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2180b1f40 0x7fe2180c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 >> 172.21.15.6:0/2250560711 conn(0x7fe2180d09b0 msgr2=0x7fe218150d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 shutdown_connections 2024-03-18T14:41:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:29.829+0000 7fe21ca7e640 1 -- 172.21.15.6:0/2250560711 wait complete. 2024-03-18T14:41:31.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:41:31.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.954+0000 7f442a977640 1 Processor -- start 2024-03-18T14:41:31.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.954+0000 7f442a977640 1 -- start start 2024-03-18T14:41:31.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f442a977640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f442415e020 0x7f4424162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f442a977640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44240d3210 0x7f44240d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f442a977640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44240b9660 con 0x7f44240ced00 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f442a977640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f44240d2270 con 0x7f44240d3210 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f442a977640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f44240ce390 con 0x7f442415e020 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f4428f14640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44240d3210 0x7f44240d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f4428f14640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44240d3210 0x7f44240d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38316/0 (socket says 172.21.15.6:38316) 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f4428f14640 1 -- 172.21.15.6:0/2032658338 learned_addr learned my addr 172.21.15.6:0/2032658338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f44237fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44240ced00 0x7f44240d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57508/0 (socket says 172.21.15.6:57508) 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.955+0000 7f4423fff640 1 --2- 172.21.15.6:0/2032658338 >> v2:172.21.15.92:3300/0 conn(0x7f442415e020 0x7f4424162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.956+0000 7f4428f14640 1 -- 172.21.15.6:0/2032658338 >> v2:172.21.15.92:3300/0 conn(0x7f442415e020 msgr2=0x7f4424162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.956+0000 7f4428f14640 1 --2- 172.21.15.6:0/2032658338 >> v2:172.21.15.92:3300/0 conn(0x7f442415e020 0x7f4424162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:31.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.956+0000 7f4428f14640 1 -- 172.21.15.6:0/2032658338 >> v1:172.21.15.6:6789/0 conn(0x7f44240ced00 legacy=0x7f44240d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:31.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.956+0000 7f4428f14640 1 -- 172.21.15.6:0/2032658338 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44240b9900 con 0x7f44240d3210 2024-03-18T14:41:31.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.956+0000 7f4422ffd640 1 -- 172.21.15.6:0/2032658338 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3977811119 0 0) 0x7f44240b9660 con 0x7f44240ced00 2024-03-18T14:41:31.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.956+0000 7f4423fff640 1 --2- 172.21.15.6:0/2032658338 >> v2:172.21.15.92:3300/0 conn(0x7f442415e020 0x7f4424162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:41:31.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.956+0000 7f4428f14640 1 --2- 172.21.15.6:0/2032658338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44240d3210 0x7f44240d3c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f44240d1450 tx=0x7f441c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42852b51dfaa84e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:31.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f4422ffd640 1 -- 172.21.15.6:0/2032658338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c02ca80 con 0x7f44240d3210 2024-03-18T14:41:31.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f4422ffd640 1 -- 172.21.15.6:0/2032658338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f441c02cc20 con 0x7f44240d3210 2024-03-18T14:41:31.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f4422ffd640 1 -- 172.21.15.6:0/2032658338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c0343f0 con 0x7f44240d3210 2024-03-18T14:41:31.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 -- 172.21.15.6:0/2032658338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44240d3210 msgr2=0x7f44240d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:31.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 --2- 172.21.15.6:0/2032658338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44240d3210 0x7f44240d3c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f44240d1450 tx=0x7f441c02bee0 comp rx=0 tx=0).stop 2024-03-18T14:41:31.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 -- 172.21.15.6:0/2032658338 shutdown_connections 2024-03-18T14:41:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 --2- 172.21.15.6:0/2032658338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44240d3210 0x7f44240d3c30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 --2- 172.21.15.6:0/2032658338 >> v2:172.21.15.92:3300/0 conn(0x7f442415e020 0x7f4424162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 -- 172.21.15.6:0/2032658338 >> 172.21.15.6:0/2032658338 conn(0x7f4424072ce0 msgr2=0x7f44240730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 -- 172.21.15.6:0/2032658338 shutdown_connections 2024-03-18T14:41:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.957+0000 7f442a977640 1 -- 172.21.15.6:0/2032658338 wait complete. 2024-03-18T14:41:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.958+0000 7f442a977640 1 Processor -- start 2024-03-18T14:41:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.958+0000 7f442a977640 1 -- start start 2024-03-18T14:41:31.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f442a977640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44240ced00 0x7f44241521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f442a977640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44240d3210 0x7f4424161a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f442a977640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f442415e020 0x7f4424161f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f442a977640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f44240cf2c0 con 0x7f442415e020 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f442a977640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f44240ce890 con 0x7f44240d3210 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f442a977640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f44240cf440 con 0x7f44240ced00 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f4428f14640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f442415e020 0x7f4424161f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f4428f14640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f442415e020 0x7f4424161f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38324/0 (socket says 172.21.15.6:38324) 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.959+0000 7f4428f14640 1 -- 172.21.15.6:0/1902026509 learned_addr learned my addr 172.21.15.6:0/1902026509 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4428f14640 1 -- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44240ced00 msgr2=0x7f44241521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4428f14640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44240ced00 0x7f44241521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f44237fe640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44240d3210 0x7f4424161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4428f14640 1 -- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44240d3210 msgr2=0x7f4424161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4428f14640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44240d3210 0x7f4424161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4428f14640 1 -- 172.21.15.6:0/1902026509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442415e400 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f44237fe640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44240d3210 0x7f4424161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4428f14640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f442415e020 0x7f4424161f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f441c036040 tx=0x7f441c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4420ff9640 1 -- 172.21.15.6:0/1902026509 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c043030 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4420ff9640 1 -- 172.21.15.6:0/1902026509 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f441c03e450 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f4420ff9640 1 -- 172.21.15.6:0/1902026509 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c009510 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442415e6e0 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.960+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442415eca0 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.961+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442408afe0 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.961+0000 7f4420ff9640 1 -- 172.21.15.6:0/1902026509 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f441c046070 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.962+0000 7f4420ff9640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f441006bbd0 0x7f441006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.962+0000 7f4420ff9640 1 -- 172.21.15.6:0/1902026509 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f441c0acb70 con 0x7f442415e020 2024-03-18T14:41:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.962+0000 7f4423fff640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f441006bbd0 0x7f441006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:31.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.962+0000 7f4423fff640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f441006bbd0 0x7f441006e090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4418006e80 tx=0x7f4418008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:31.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:31.966+0000 7f4420ff9640 1 -- 172.21.15.6:0/1902026509 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f441c04d070 con 0x7f442415e020 2024-03-18T14:41:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.189+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f442404f620 con 0x7f442415e020 2024-03-18T14:41:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.190+0000 7f4420ff9640 1 -- 172.21.15.6:0/1902026509 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3855 (secure 0 0 0) 0x7f441c0333a0 con 0x7f442415e020 2024-03-18T14:41:32.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:32.192 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T14:41:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 3 2024-03-18T14:41:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f441006bbd0 msgr2=0x7f441006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f441006bbd0 0x7f441006e090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4418006e80 tx=0x7f4418008040 comp rx=0 tx=0).stop 2024-03-18T14:41:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f442415e020 msgr2=0x7f4424161f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f442415e020 0x7f4424161f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f441c036040 tx=0x7f441c004a40 comp rx=0 tx=0).stop 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 shutdown_connections 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f441006bbd0 0x7f441006e090 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f442415e020 0x7f4424161f60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44240d3210 0x7f4424161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 --2- 172.21.15.6:0/1902026509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44240ced00 0x7f44241521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 >> 172.21.15.6:0/1902026509 conn(0x7f4424072ce0 msgr2=0x7f44240d2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 shutdown_connections 2024-03-18T14:41:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.192+0000 7f442a977640 1 -- 172.21.15.6:0/1902026509 wait complete. 2024-03-18T14:41:32.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:32.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98c0c36640 1 Processor -- start 2024-03-18T14:41:32.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98c0c36640 1 -- start start 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98c0c36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f98bc15bd40 0x7f98bc0d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98c0c36640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98c0c36640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98bc0cc870 con 0x7f98bc0d1d90 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98c0c36640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f98bc0d0500 con 0x7f98bc15bd40 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98c0c36640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f98bc0d07e0 con 0x7f98bc0cfea0 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98ba59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f98bc15bd40 0x7f98bc0d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98b3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f98bc0d1d90 0x7f98bc0cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57510/0 (socket says 172.21.15.6:57510) 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98b3fff640 1 -- 172.21.15.6:0/1533495271 learned_addr learned my addr 172.21.15.6:0/1533495271 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.305+0000 7f98ba59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f98bc15bd40 0x7f98bc0d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38330/0 (socket says 172.21.15.6:38330) 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.306+0000 7f98bad9d640 1 --2- 172.21.15.6:0/1533495271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.306+0000 7f98ba59c640 1 -- 172.21.15.6:0/1533495271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 msgr2=0x7f98bc0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:32.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.306+0000 7f98ba59c640 1 --2- 172.21.15.6:0/1533495271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.306+0000 7f98ba59c640 1 -- 172.21.15.6:0/1533495271 >> v1:172.21.15.6:6789/0 conn(0x7f98bc0d1d90 legacy=0x7f98bc0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.306+0000 7f98ba59c640 1 -- 172.21.15.6:0/1533495271 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98bc0ccb10 con 0x7f98bc15bd40 2024-03-18T14:41:32.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.306+0000 7f98b9d9b640 1 -- 172.21.15.6:0/1533495271 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330167100 0 0) 0x7f98bc0cc870 con 0x7f98bc0d1d90 2024-03-18T14:41:32.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.306+0000 7f98ba59c640 1 --2- 172.21.15.6:0/1533495271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0d1850 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0097a0 tx=0x7f98ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb180e974488a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98b9d9b640 1 -- 172.21.15.6:0/1533495271 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac035020 con 0x7f98bc15bd40 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98b9d9b640 1 -- 172.21.15.6:0/1533495271 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ac031830 con 0x7f98bc15bd40 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98b9d9b640 1 -- 172.21.15.6:0/1533495271 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac031c00 con 0x7f98bc15bd40 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 -- 172.21.15.6:0/1533495271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 msgr2=0x7f98bc0d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1533495271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0d1850 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0097a0 tx=0x7f98ac02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 -- 172.21.15.6:0/1533495271 shutdown_connections 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1533495271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1533495271 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0d1850 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 -- 172.21.15.6:0/1533495271 >> 172.21.15.6:0/1533495271 conn(0x7f98bc09ae20 msgr2=0x7f98bc08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 -- 172.21.15.6:0/1533495271 shutdown_connections 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.307+0000 7f98c0c36640 1 -- 172.21.15.6:0/1533495271 wait complete. 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 Processor -- start 2024-03-18T14:41:32.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 -- start start 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98bc0d1d90 0x7f98bc14df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc0d0c40 con 0x7f98bc15bd40 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc0d0ac0 con 0x7f98bc0d1d90 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98c0c36640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f98bc15c120 con 0x7f98bc0cfea0 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98bad9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98bad9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0b63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38334/0 (socket says 172.21.15.6:38334) 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.309+0000 7f98bad9d640 1 -- 172.21.15.6:0/1110236302 learned_addr learned my addr 172.21.15.6:0/1110236302 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98ba59c640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98b3fff640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98bc0d1d90 0x7f98bc14df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98bad9d640 1 -- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 msgr2=0x7f98bc0b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98bad9d640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98bad9d640 1 -- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98bc0d1d90 msgr2=0x7f98bc14df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98bad9d640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98bc0d1d90 0x7f98bc14df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98bad9d640 1 -- 172.21.15.6:0/1110236302 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98bc15d300 con 0x7f98bc15bd40 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98ba59c640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98bad9d640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0b63d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f98b4007e40 tx=0x7f98b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98b2ffd640 1 -- 172.21.15.6:0/1110236302 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4014620 con 0x7f98bc15bd40 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98b2ffd640 1 -- 172.21.15.6:0/1110236302 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b4019070 con 0x7f98bc15bd40 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98b2ffd640 1 -- 172.21.15.6:0/1110236302 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4014a20 con 0x7f98bc15bd40 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98bc0b4ce0 con 0x7f98bc15bd40 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.310+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98bc0b52a0 con 0x7f98bc15bd40 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.311+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98bc09fa90 con 0x7f98bc15bd40 2024-03-18T14:41:32.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.312+0000 7f98b2ffd640 1 -- 172.21.15.6:0/1110236302 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f98b4014bc0 con 0x7f98bc15bd40 2024-03-18T14:41:32.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.312+0000 7f98b2ffd640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f989c06bbd0 0x7f989c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.312+0000 7f98b2ffd640 1 -- 172.21.15.6:0/1110236302 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f98b408b060 con 0x7f98bc15bd40 2024-03-18T14:41:32.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.312+0000 7f98ba59c640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f989c06bbd0 0x7f989c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.313+0000 7f98ba59c640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f989c06bbd0 0x7f989c06e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98ac036000 tx=0x7f98ac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.316+0000 7f98b2ffd640 1 -- 172.21.15.6:0/1110236302 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f98bc15d300 con 0x7f98bc15bd40 2024-03-18T14:41:32.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.503+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98bc04f620 con 0x7f98bc15bd40 2024-03-18T14:41:32.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.503+0000 7f98b2ffd640 1 -- 172.21.15.6:0/1110236302 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12788 (secure 0 0 0) 0x7f98b4059ea0 con 0x7f98bc15bd40 2024-03-18T14:41:32.505 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:32.506 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:26.411799+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f989c06bbd0 msgr2=0x7f989c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f989c06bbd0 0x7f989c06e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98ac036000 tx=0x7f98ac008030 comp rx=0 tx=0).stop 2024-03-18T14:41:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 msgr2=0x7f98bc0b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0b63d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f98b4007e40 tx=0x7f98b400e4b0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 shutdown_connections 2024-03-18T14:41:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f989c06bbd0 0x7f989c06e090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98bc15bd40 0x7f98bc0b63d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98bc0d1d90 0x7f98bc14df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 --2- 172.21.15.6:0/1110236302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98bc0cfea0 0x7f98bc0b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 >> 172.21.15.6:0/1110236302 conn(0x7f98bc09ae20 msgr2=0x7f98bc0714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 shutdown_connections 2024-03-18T14:41:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.506+0000 7f98c0c36640 1 -- 172.21.15.6:0/1110236302 wait complete. 2024-03-18T14:41:32.518 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:41:32.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:41:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.621+0000 7fe34b46c640 1 Processor -- start 2024-03-18T14:41:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.621+0000 7fe34b46c640 1 -- start start 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe34b46c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe34415dfd0 0x7fe3440b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe34b46c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3440b7e00 0x7fe3440b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe34b46c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34415e3b0 con 0x7fe3440b5740 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe34b46c640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe3440b8470 con 0x7fe34415dfd0 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe34b46c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3440b8750 con 0x7fe3440b7e00 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349208640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe34415dfd0 0x7fe3440b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349208640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe34415dfd0 0x7fe3440b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38344/0 (socket says 172.21.15.6:38344) 2024-03-18T14:41:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349208640 1 -- 172.21.15.6:0/835881602 learned_addr learned my addr 172.21.15.6:0/835881602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349a09640 1 --2- 172.21.15.6:0/835881602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3440b7e00 0x7fe3440b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349208640 1 -- 172.21.15.6:0/835881602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3440b7e00 msgr2=0x7fe3440b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349208640 1 --2- 172.21.15.6:0/835881602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3440b7e00 0x7fe3440b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349208640 1 -- 172.21.15.6:0/835881602 >> v1:172.21.15.92:6789/0 conn(0x7fe3440b5740 legacy=0x7fe3440d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.622+0000 7fe349208640 1 -- 172.21.15.6:0/835881602 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3440b8a30 con 0x7fe34415dfd0 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.623+0000 7fe349208640 1 --2- 172.21.15.6:0/835881602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe34415dfd0 0x7fe3440b9650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009870 tx=0x7fe33c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562bd7c9d81a1d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.623+0000 7fe343fff640 1 -- 172.21.15.6:0/835881602 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c034070 con 0x7fe34415dfd0 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.623+0000 7fe343fff640 1 -- 172.21.15.6:0/835881602 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe33c030840 con 0x7fe34415dfd0 2024-03-18T14:41:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.623+0000 7fe343fff640 1 -- 172.21.15.6:0/835881602 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c030c10 con 0x7fe34415dfd0 2024-03-18T14:41:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 -- 172.21.15.6:0/835881602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe34415dfd0 msgr2=0x7fe3440b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 --2- 172.21.15.6:0/835881602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe34415dfd0 0x7fe3440b9650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009870 tx=0x7fe33c02b980 comp rx=0 tx=0).stop 2024-03-18T14:41:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 -- 172.21.15.6:0/835881602 shutdown_connections 2024-03-18T14:41:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 --2- 172.21.15.6:0/835881602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3440b7e00 0x7fe3440b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 --2- 172.21.15.6:0/835881602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe34415dfd0 0x7fe3440b9650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 -- 172.21.15.6:0/835881602 >> 172.21.15.6:0/835881602 conn(0x7fe344072d10 msgr2=0x7fe344073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 -- 172.21.15.6:0/835881602 shutdown_connections 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.624+0000 7fe34b46c640 1 -- 172.21.15.6:0/835881602 wait complete. 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.625+0000 7fe34b46c640 1 Processor -- start 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.625+0000 7fe34b46c640 1 -- start start 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe34b46c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440b5740 0x7fe3441f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe34b46c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3440b7e00 0x7fe3440ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe34b46c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe34415dfd0 0x7fe344160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe34b46c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe3440d2550 con 0x7fe3440b5740 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe34b46c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe3440b9b90 con 0x7fe3440b7e00 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe34b46c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3440d26d0 con 0x7fe34415dfd0 2024-03-18T14:41:32.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440b5740 0x7fe3441f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440b5740 0x7fe3441f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38352/0 (socket says 172.21.15.6:38352) 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 -- 172.21.15.6:0/2235012357 learned_addr learned my addr 172.21.15.6:0/2235012357 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349a09640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe34415dfd0 0x7fe344160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe348a07640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3440b7e00 0x7fe3440ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 -- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe34415dfd0 msgr2=0x7fe344160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe34415dfd0 0x7fe344160460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 -- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3440b7e00 msgr2=0x7fe3440ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3440b7e00 0x7fe3440ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 -- 172.21.15.6:0/2235012357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe344160bb0 con 0x7fe3440b5740 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe349208640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440b5740 0x7fe3441f3d10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe33c000c00 tx=0x7fe33c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.626+0000 7fe341ffb640 1 -- 172.21.15.6:0/2235012357 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c034040 con 0x7fe3440b5740 2024-03-18T14:41:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.627+0000 7fe341ffb640 1 -- 172.21.15.6:0/2235012357 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe33c039d90 con 0x7fe3440b5740 2024-03-18T14:41:32.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.627+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe344160e90 con 0x7fe3440b5740 2024-03-18T14:41:32.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.627+0000 7fe341ffb640 1 -- 172.21.15.6:0/2235012357 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c043630 con 0x7fe3440b5740 2024-03-18T14:41:32.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.627+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe34415e9b0 con 0x7fe3440b5740 2024-03-18T14:41:32.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.628+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3440751b0 con 0x7fe3440b5740 2024-03-18T14:41:32.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.628+0000 7fe341ffb640 1 -- 172.21.15.6:0/2235012357 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe33c04a060 con 0x7fe3440b5740 2024-03-18T14:41:32.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.629+0000 7fe341ffb640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe33006bbd0 0x7fe33006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.629+0000 7fe341ffb640 1 -- 172.21.15.6:0/2235012357 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fe33c0ac660 con 0x7fe3440b5740 2024-03-18T14:41:32.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.629+0000 7fe348a07640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe33006bbd0 0x7fe33006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.629+0000 7fe348a07640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe33006bbd0 0x7fe33006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe3440aa820 tx=0x7fe338009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.633+0000 7fe341ffb640 1 -- 172.21.15.6:0/2235012357 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe33c03e070 con 0x7fe3440b5740 2024-03-18T14:41:32.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.814+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe3440b3160 con 0x7fe3440b5740 2024-03-18T14:41:32.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.814+0000 7fe341ffb640 1 -- 172.21.15.6:0/2235012357 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7fe33c02d330 con 0x7fe3440b5740 2024-03-18T14:41:32.816 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:41:32.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe33006bbd0 msgr2=0x7fe33006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe33006bbd0 0x7fe33006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe3440aa820 tx=0x7fe338009040 comp rx=0 tx=0).stop 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440b5740 msgr2=0x7fe3441f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440b5740 0x7fe3441f3d10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe33c000c00 tx=0x7fe33c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 shutdown_connections 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe33006bbd0 0x7fe33006e090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe34415dfd0 0x7fe344160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3440b7e00 0x7fe3440ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 --2- 172.21.15.6:0/2235012357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440b5740 0x7fe3441f3d10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 >> 172.21.15.6:0/2235012357 conn(0x7fe344072d10 msgr2=0x7fe3440ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 shutdown_connections 2024-03-18T14:41:32.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.816+0000 7fe34b46c640 1 -- 172.21.15.6:0/2235012357 wait complete. 2024-03-18T14:41:32.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T14:41:32.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.929+0000 7fa3f1937640 1 Processor -- start 2024-03-18T14:41:32.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.929+0000 7fa3f1937640 1 -- start start 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.929+0000 7fa3f1937640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3ec06d3f0 0x7fa3ec0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.929+0000 7fa3f1937640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec0b28b0 con 0x7fa3ec15def0 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.929+0000 7fa3f1937640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec0b2b90 con 0x7fa3ec0b1b50 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.929+0000 7fa3f1937640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa3ec0d0540 con 0x7fa3ec06d3f0 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3eaffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3ec15def0 0x7fa3ec06cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57524/0 (socket says 172.21.15.6:57524) 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3eaffd640 1 -- 172.21.15.6:0/3461025341 learned_addr learned my addr 172.21.15.6:0/3461025341 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3e3fff640 1 --2- 172.21.15.6:0/3461025341 >> v2:172.21.15.92:3300/0 conn(0x7fa3ec06d3f0 0x7fa3ec0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4294859369 0 0) 0x7fa3ec0b28b0 con 0x7fa3ec15def0 2024-03-18T14:41:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d8003520 con 0x7fa3ec15def0 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3236147086 0 0) 0x7fa3d8003520 con 0x7fa3ec15def0 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 >> v2:172.21.15.92:3300/0 conn(0x7fa3ec06d3f0 msgr2=0x7fa3ec0cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 --2- 172.21.15.6:0/3461025341 >> v2:172.21.15.92:3300/0 conn(0x7fa3ec06d3f0 0x7fa3ec0cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 >> v1:172.21.15.92:6789/0 conn(0x7fa3ec0b1b50 legacy=0x7fa3ec0b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec0d01d0 con 0x7fa3ec15def0 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fa3dc001670 con 0x7fa3ec15def0 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 >> v1:172.21.15.6:6789/0 conn(0x7fa3ec15def0 legacy=0x7fa3ec06cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d80039b0 0x7fa3d8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80043d0 0x7fa3d80048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80050a0 0x7fa3d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ec0d0540 con 0x7fa3d80039b0 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d800bfe0 con 0x7fa3d80050a0 2024-03-18T14:41:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d800c1e0 con 0x7fa3d80043d0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3eaffd640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d80039b0 0x7fa3d8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3eb7fe640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80050a0 0x7fa3d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.930+0000 7fa3e3fff640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80043d0 0x7fa3d80048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3eaffd640 1 -- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80043d0 msgr2=0x7fa3d80048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3eaffd640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80043d0 0x7fa3d80048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3eaffd640 1 -- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80050a0 msgr2=0x7fa3d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3eaffd640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d80050a0 0x7fa3d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3eaffd640 1 -- 172.21.15.6:0/3461025341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d8003520 con 0x7fa3d80039b0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3eb7fe640 1 -- 172.21.15.6:0/3461025341 reap_dead start 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3eaffd640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d80039b0 0x7fa3d8003db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00c240 tx=0x7fa3d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef752ef62eed8bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc007d10 con 0x7fa3d80039b0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3dc030070 con 0x7fa3d80039b0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.931+0000 7fa3ea7fc640 1 -- 172.21.15.6:0/3461025341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc02cda0 con 0x7fa3d80039b0 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.932+0000 7fa3f1937640 1 -- 172.21.15.6:0/3461025341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d80039b0 msgr2=0x7fa3d8003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.932+0000 7fa3f1937640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d80039b0 0x7fa3d8003db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00c240 tx=0x7fa3d8005830 comp rx=0 tx=0).stop 2024-03-18T14:41:32.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.932+0000 7fa3f1937640 1 -- 172.21.15.6:0/3461025341 shutdown_connections 2024-03-18T14:41:32.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.932+0000 7fa3f1937640 1 --2- 172.21.15.6:0/3461025341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3d80039b0 0x7fa3d8003db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.932+0000 7fa3f1937640 1 -- 172.21.15.6:0/3461025341 >> 172.21.15.6:0/3461025341 conn(0x7fa3ec09ae20 msgr2=0x7fa3ec08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:32.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.932+0000 7fa3f1937640 1 -- 172.21.15.6:0/3461025341 shutdown_connections 2024-03-18T14:41:32.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.932+0000 7fa3f1937640 1 -- 172.21.15.6:0/3461025341 wait complete. 2024-03-18T14:41:32.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.933+0000 7fa3f1937640 1 Processor -- start 2024-03-18T14:41:32.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.933+0000 7fa3f1937640 1 -- start start 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3f1937640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80039b0 0x7fa3ec15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3f1937640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec160b60 0x7fa3ec1f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3f1937640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec160fe0 0x7fa3ec1f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3f1937640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ec0cc1a0 con 0x7fa3ec160b60 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3f1937640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ec15e2d0 con 0x7fa3ec160fe0 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3f1937640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ec0cc320 con 0x7fa3d80039b0 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec160b60 0x7fa3ec1f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec160b60 0x7fa3ec1f6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38370/0 (socket says 172.21.15.6:38370) 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 -- 172.21.15.6:0/2018160517 learned_addr learned my addr 172.21.15.6:0/2018160517 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3eaffd640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80039b0 0x7fa3ec15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3eb7fe640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec160fe0 0x7fa3ec1f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 -- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80039b0 msgr2=0x7fa3ec15bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80039b0 0x7fa3ec15bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 -- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec160fe0 msgr2=0x7fa3ec1f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec160fe0 0x7fa3ec1f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 -- 172.21.15.6:0/2018160517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec15e450 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e3fff640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec160b60 0x7fa3ec1f6cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4011990 tx=0x7fa3d4011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.934+0000 7fa3e8ff9640 1 -- 172.21.15.6:0/2018160517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d400fad0 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.935+0000 7fa3e8ff9640 1 -- 172.21.15.6:0/2018160517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3d400fc70 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.935+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ec15e730 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.935+0000 7fa3e8ff9640 1 -- 172.21.15.6:0/2018160517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4014440 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.935+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3ec15ecf0 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.936+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3ec09fa90 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.936+0000 7fa3e8ff9640 1 -- 172.21.15.6:0/2018160517 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3d4012040 con 0x7fa3ec160b60 2024-03-18T14:41:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.936+0000 7fa3e8ff9640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.936+0000 7fa3e8ff9640 1 -- 172.21.15.6:0/2018160517 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa3d408acf0 con 0x7fa3ec160b60 2024-03-18T14:41:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.936+0000 7fa3eaffd640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.937+0000 7fa3eaffd640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc034000 tx=0x7fa3dc007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:32.941+0000 7fa3e8ff9640 1 -- 172.21.15.6:0/2018160517 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3d408f070 con 0x7fa3ec160b60 2024-03-18T14:41:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.126+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa3ec04f620 con 0x7fa3ec160b60 2024-03-18T14:41:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.126+0000 7fa3e8ff9640 1 -- 172.21.15.6:0/2018160517 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fa3d4019030 con 0x7fa3ec160b60 2024-03-18T14:41:33.128 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:33.128 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-18T14:41:33.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:41:33.128 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-18T14:41:33.128 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-18T14:41:33.129 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:41:33.129 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 msgr2=0x7fa3cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc034000 tx=0x7fa3dc007000 comp rx=0 tx=0).stop 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec160b60 msgr2=0x7fa3ec1f6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec160b60 0x7fa3ec1f6cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4011990 tx=0x7fa3d4011e50 comp rx=0 tx=0).stop 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 shutdown_connections 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec160fe0 0x7fa3ec1f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec160b60 0x7fa3ec1f6cd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 --2- 172.21.15.6:0/2018160517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d80039b0 0x7fa3ec15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 >> 172.21.15.6:0/2018160517 conn(0x7fa3ec09ae20 msgr2=0x7fa3ec15d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:33.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 shutdown_connections 2024-03-18T14:41:33.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.128+0000 7fa3f1937640 1 -- 172.21.15.6:0/2018160517 wait complete. 2024-03-18T14:41:33.140 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T14:41:33.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T14:41:33.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.244+0000 7f1079d10640 1 Processor -- start 2024-03-18T14:41:33.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.244+0000 7f1079d10640 1 -- start start 2024-03-18T14:41:33.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f1079d10640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 0x7f10740cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f1079d10640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10740ccb20 con 0x7f10740cc600 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f1079d10640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10740ccdc0 con 0x7f10740cf6e0 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f1079d10640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f10740c45f0 con 0x7f107415adb0 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f1072ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f10740cc600 0x7f10740cefd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57534/0 (socket says 172.21.15.6:57534) 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f1072ffd640 1 -- 172.21.15.6:0/2777400381 learned_addr learned my addr 172.21.15.6:0/2777400381 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f10737fe640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 0x7f10740cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453830142 0 0) 0x7f10740ccb20 con 0x7f10740cc600 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.245+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f105c003520 con 0x7f10740cc600 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 173139294 0 0) 0x7f105c003520 con 0x7f10740cc600 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 msgr2=0x7f10740cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 0x7f10740cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 >> v1:172.21.15.92:6789/0 conn(0x7f10740cf6e0 legacy=0x7f10740c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10740cd060 con 0x7f10740cc600 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f1068001690 con 0x7f10740cc600 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 >> v1:172.21.15.6:6789/0 conn(0x7f10740cc600 legacy=0x7f10740cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 0x7f105c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c003f80 0x7f105c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f105c004d40 0x7f105c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f10740c45f0 con 0x7f105c004d40 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f105c00bb80 con 0x7f105c003f80 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f105c00bd80 con 0x7f107415adb0 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f1073fff640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f105c004d40 0x7f105c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f10737fe640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 0x7f105c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f1072ffd640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c003f80 0x7f105c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f1073fff640 1 -- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 msgr2=0x7f105c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:33.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f1073fff640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f107415adb0 0x7f105c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f1073fff640 1 -- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c003f80 msgr2=0x7f105c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f1073fff640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f105c003f80 0x7f105c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.246+0000 7f1073fff640 1 -- 172.21.15.6:0/2777400381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c003520 con 0x7f105c004d40 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1073fff640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f105c004d40 0x7f105c0051c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f10740ce470 tx=0x7f105c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b85edf3fda40a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c001a70 con 0x7f105c004d40 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f106c003b70 con 0x7f105c004d40 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f10727fc640 1 -- 172.21.15.6:0/2777400381 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c0050d0 con 0x7f105c004d40 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1079d10640 1 -- 172.21.15.6:0/2777400381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f105c004d40 msgr2=0x7f105c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1079d10640 1 --2- 172.21.15.6:0/2777400381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f105c004d40 0x7f105c0051c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f10740ce470 tx=0x7f105c005830 comp rx=0 tx=0).stop 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1073fff640 1 -- 172.21.15.6:0/2777400381 reap_dead start 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1079d10640 1 -- 172.21.15.6:0/2777400381 shutdown_connections 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1079d10640 1 -- 172.21.15.6:0/2777400381 >> 172.21.15.6:0/2777400381 conn(0x7f1074099a50 msgr2=0x7f1074099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:33.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1079d10640 1 -- 172.21.15.6:0/2777400381 shutdown_connections 2024-03-18T14:41:33.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.247+0000 7f1079d10640 1 -- 172.21.15.6:0/2777400381 wait complete. 2024-03-18T14:41:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 Processor -- start 2024-03-18T14:41:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 -- start start 2024-03-18T14:41:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10741571c0 0x7f10740c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10740c9250 0x7f107415c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10740ca780 0x7f107415cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f10740d0a10 con 0x7f10740c9250 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10740c48d0 con 0x7f10740ca780 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.249+0000 7f1079d10640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f10740cc0c0 con 0x7f10741571c0 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10740c9250 0x7f107415c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10740c9250 0x7f107415c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38400/0 (socket says 172.21.15.6:38400) 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 -- 172.21.15.6:0/389484142 learned_addr learned my addr 172.21.15.6:0/389484142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f10737fe640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10741571c0 0x7f10740c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 -- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10741571c0 msgr2=0x7f10740c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10741571c0 0x7f10740c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 -- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10740ca780 msgr2=0x7f107415cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10740ca780 0x7f107415cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 -- 172.21.15.6:0/389484142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10740c96e0 con 0x7f10740c9250 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1072ffd640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10740c9250 0x7f107415c650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1068007310 tx=0x7f10680077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f105bfff640 1 -- 172.21.15.6:0/389484142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106800f8c0 con 0x7f10740c9250 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f105bfff640 1 -- 172.21.15.6:0/389484142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1068018070 con 0x7f10740c9250 2024-03-18T14:41:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10740c99c0 con 0x7f10740c9250 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f105bfff640 1 -- 172.21.15.6:0/389484142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106800fce0 con 0x7f10740c9250 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.250+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10740c9f80 con 0x7f10740c9250 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.251+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f107409eaf0 con 0x7f10740c9250 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.252+0000 7f105bfff640 1 -- 172.21.15.6:0/389484142 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f106801d460 con 0x7f10740c9250 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.252+0000 7f105bfff640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f106006bbd0 0x7f106006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.252+0000 7f105bfff640 1 -- 172.21.15.6:0/389484142 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f106808a800 con 0x7f10740c9250 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.252+0000 7f10737fe640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f106006bbd0 0x7f106006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.253+0000 7f10737fe640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f106006bbd0 0x7f106006e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f10640099e0 tx=0x7f1064008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:33.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.256+0000 7f105bfff640 1 -- 172.21.15.6:0/389484142 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1068058ff0 con 0x7f10740c9250 2024-03-18T14:41:33.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.439+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f107404f620 con 0x7f10740c9250 2024-03-18T14:41:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.509+0000 7f105bfff640 1 -- 172.21.15.6:0/389484142 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f106805d090 con 0x7f10740c9250 2024-03-18T14:41:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-18T14:41:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f106006bbd0 msgr2=0x7f106006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f106006bbd0 0x7f106006e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f10640099e0 tx=0x7f1064008040 comp rx=0 tx=0).stop 2024-03-18T14:41:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10740c9250 msgr2=0x7f107415c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10740c9250 0x7f107415c650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1068007310 tx=0x7f10680077d0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 shutdown_connections 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f106006bbd0 0x7f106006e090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10740ca780 0x7f107415cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10740c9250 0x7f107415c650 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 --2- 172.21.15.6:0/389484142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10741571c0 0x7f10740c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 >> 172.21.15.6:0/389484142 conn(0x7f1074099a50 msgr2=0x7f10740c5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 shutdown_connections 2024-03-18T14:41:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.511+0000 7f1079d10640 1 -- 172.21.15.6:0/389484142 wait complete. 2024-03-18T14:41:33.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T14:41:33.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.633+0000 7fb11c0f1640 1 Processor -- start 2024-03-18T14:41:33.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.633+0000 7fb11c0f1640 1 -- start start 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.634+0000 7fb11c0f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 0x7fb11415e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.634+0000 7fb11c0f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11415edd0 con 0x7fb11415e950 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.634+0000 7fb11c0f1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11415f0b0 con 0x7fb1140ce970 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.634+0000 7fb11c0f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb1140d21a0 con 0x7fb1140b9940 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.634+0000 7fb11a68e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb11415e950 0x7fb1140d2630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42892/0 (socket says 172.21.15.6:42892) 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.634+0000 7fb11a68e640 1 -- 172.21.15.6:0/3795349969 learned_addr learned my addr 172.21.15.6:0/3795349969 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.634+0000 7fb11968c640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 0x7fb11415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426800092 0 0) 0x7fb11415edd0 con 0x7fb11415e950 2024-03-18T14:41:33.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0fc003520 con 0x7fb11415e950 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4142127873 0 0) 0x7fb0fc003520 con 0x7fb11415e950 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 msgr2=0x7fb11415e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 0x7fb11415e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 >> v1:172.21.15.92:6789/0 conn(0x7fb1140ce970 legacy=0x7fb1140ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1140d1e30 con 0x7fb11415e950 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fb10c001670 con 0x7fb11415e950 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 >> v1:172.21.15.6:6789/0 conn(0x7fb11415e950 legacy=0x7fb1140d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0fc0039e0 0x7fb0fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 0x7fb0fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0fc004d40 0x7fb0fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb1140d21a0 con 0x7fb0fc0039e0 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc00b9c0 con 0x7fb0fc004d40 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb0fc00bbc0 con 0x7fb1140b9940 2024-03-18T14:41:33.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb119e8d640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0fc0039e0 0x7fb0fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb11968c640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 0x7fb0fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb11a68e640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0fc004d40 0x7fb0fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb119e8d640 1 -- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 msgr2=0x7fb0fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb119e8d640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1140b9940 0x7fb0fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb119e8d640 1 -- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0fc004d40 msgr2=0x7fb0fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb119e8d640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0fc004d40 0x7fb0fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb119e8d640 1 -- 172.21.15.6:0/3795349969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc003520 con 0x7fb0fc0039e0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.635+0000 7fb11a68e640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0fc004d40 0x7fb0fc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.636+0000 7fb119e8d640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0fc0039e0 0x7fb0fc003dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb104002010 tx=0x7fb0fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3f5649dd772f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.636+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104001be0 con 0x7fb0fc0039e0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.636+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb10402d6c0 con 0x7fb0fc0039e0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.636+0000 7fb118e8b640 1 -- 172.21.15.6:0/3795349969 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10402da90 con 0x7fb0fc0039e0 2024-03-18T14:41:33.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.637+0000 7fb11c0f1640 1 -- 172.21.15.6:0/3795349969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0fc0039e0 msgr2=0x7fb0fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.637+0000 7fb11c0f1640 1 --2- 172.21.15.6:0/3795349969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0fc0039e0 0x7fb0fc003dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb104002010 tx=0x7fb0fc005830 comp rx=0 tx=0).stop 2024-03-18T14:41:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.637+0000 7fb11a68e640 1 -- 172.21.15.6:0/3795349969 reap_dead start 2024-03-18T14:41:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.637+0000 7fb11c0f1640 1 -- 172.21.15.6:0/3795349969 shutdown_connections 2024-03-18T14:41:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.637+0000 7fb11c0f1640 1 -- 172.21.15.6:0/3795349969 >> 172.21.15.6:0/3795349969 conn(0x7fb11406c900 msgr2=0x7fb11406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.637+0000 7fb11c0f1640 1 -- 172.21.15.6:0/3795349969 shutdown_connections 2024-03-18T14:41:33.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.637+0000 7fb11c0f1640 1 -- 172.21.15.6:0/3795349969 wait complete. 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.638+0000 7fb11c0f1640 1 Processor -- start 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.638+0000 7fb11c0f1640 1 -- start start 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0fc0039e0 0x7fb1140b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1140b4aa0 0x7fb11415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1140b1e10 0x7fb1140ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb11415bd70 con 0x7fb1140b1e10 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb11415e410 con 0x7fb1140b4aa0 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb1140b4fa0 con 0x7fb0fc0039e0 2024-03-18T14:41:33.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1140b1e10 0x7fb1140ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1140b1e10 0x7fb1140ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42856/0 (socket says 172.21.15.6:42856) 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 -- 172.21.15.6:0/2366291845 learned_addr learned my addr 172.21.15.6:0/2366291845 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11968c640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1140b4aa0 0x7fb11415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb119e8d640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0fc0039e0 0x7fb1140b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 -- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0fc0039e0 msgr2=0x7fb1140b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0fc0039e0 0x7fb1140b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 -- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1140b4aa0 msgr2=0x7fb11415a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1140b4aa0 0x7fb11415a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 -- 172.21.15.6:0/2366291845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1140b5280 con 0x7fb1140b1e10 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11a68e640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1140b1e10 0x7fb1140ccf40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1140b36b0 tx=0x7fb10c00ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb1127fc640 1 -- 172.21.15.6:0/2366291845 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c010510 con 0x7fb1140b1e10 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb1127fc640 1 -- 172.21.15.6:0/2366291845 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb10c0106b0 con 0x7fb1140b1e10 2024-03-18T14:41:33.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1140b5560 con 0x7fb1140b1e10 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb1127fc640 1 -- 172.21.15.6:0/2366291845 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c010a60 con 0x7fb1140b1e10 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.639+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1140b5b20 con 0x7fb1140b1e10 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.640+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1140b5d20 con 0x7fb1140b1e10 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.641+0000 7fb1127fc640 1 -- 172.21.15.6:0/2366291845 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb10c010c00 con 0x7fb1140b1e10 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.641+0000 7fb1127fc640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb10006bc00 0x7fb10006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.642+0000 7fb1127fc640 1 -- 172.21.15.6:0/2366291845 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fb10c089eb0 con 0x7fb1140b1e10 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.642+0000 7fb119e8d640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb10006bc00 0x7fb10006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.642+0000 7fb119e8d640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb10006bc00 0x7fb10006e0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb104036000 tx=0x7fb104009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:33.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.646+0000 7fb1127fc640 1 -- 172.21.15.6:0/2366291845 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb1140b5d20 con 0x7fb1140b1e10 2024-03-18T14:41:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:33.829+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fb1140751b0 con 0x7fb1140b1e10 2024-03-18T14:41:34.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.511+0000 7fb1127fc640 1 -- 172.21.15.6:0/2366291845 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7fb10c0584a0 con 0x7fb1140b1e10 2024-03-18T14:41:34.513 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-18T14:41:34.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.514+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb10006bc00 msgr2=0x7fb10006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.514+0000 7fb11c0f1640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb10006bc00 0x7fb10006e0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb104036000 tx=0x7fb104009030 comp rx=0 tx=0).stop 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.514+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1140b1e10 msgr2=0x7fb1140ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.514+0000 7fb11c0f1640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1140b1e10 0x7fb1140ccf40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1140b36b0 tx=0x7fb10c00ead0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 shutdown_connections 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb10006bc00 0x7fb10006e0c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1140b1e10 0x7fb1140ccf40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1140b4aa0 0x7fb11415a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 --2- 172.21.15.6:0/2366291845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0fc0039e0 0x7fb1140b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 >> 172.21.15.6:0/2366291845 conn(0x7fb11406c900 msgr2=0x7fb114160b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 shutdown_connections 2024-03-18T14:41:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.515+0000 7fb11c0f1640 1 -- 172.21.15.6:0/2366291845 wait complete. 2024-03-18T14:41:34.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T14:41:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.650+0000 7f4197c36640 1 Processor -- start 2024-03-18T14:41:34.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.650+0000 7f4197c36640 1 -- start start 2024-03-18T14:41:34.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f4197c36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41900d2e50 0x7f41900b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:34.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f4197c36640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f41900b5c80 0x7f41900b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:34.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f4197c36640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419015e3b0 con 0x7f419015dfd0 2024-03-18T14:41:34.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f4197c36640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f41900b8470 con 0x7f41900d2e50 2024-03-18T14:41:34.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f4197c36640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f41900b8750 con 0x7f41900b5c80 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41951d1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41900d2e50 0x7f41900b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41959d2640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f419015dfd0 0x7f41900b9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42904/0 (socket says 172.21.15.6:42904) 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41951d1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41900d2e50 0x7f41900b5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42862/0 (socket says 172.21.15.6:42862) 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41959d2640 1 -- 172.21.15.6:0/411604108 learned_addr learned my addr 172.21.15.6:0/411604108 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41961d3640 1 --2- 172.21.15.6:0/411604108 >> v2:172.21.15.92:3300/0 conn(0x7f41900b5c80 0x7f41900b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41951d1640 1 -- 172.21.15.6:0/411604108 >> v2:172.21.15.92:3300/0 conn(0x7f41900b5c80 msgr2=0x7f41900b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41951d1640 1 --2- 172.21.15.6:0/411604108 >> v2:172.21.15.92:3300/0 conn(0x7f41900b5c80 0x7f41900b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41951d1640 1 -- 172.21.15.6:0/411604108 >> v1:172.21.15.6:6789/0 conn(0x7f419015dfd0 legacy=0x7f41900b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41951d1640 1 -- 172.21.15.6:0/411604108 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41900b8a30 con 0x7f41900d2e50 2024-03-18T14:41:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.651+0000 7f41949d0640 1 -- 172.21.15.6:0/411604108 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4196934974 0 0) 0x7f419015e3b0 con 0x7f419015dfd0 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.652+0000 7f41951d1640 1 --2- 172.21.15.6:0/411604108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900d2e50 0x7f41900b5740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f41900d16f0 tx=0x7f418002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6993bf609fa567ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.652+0000 7f41949d0640 1 -- 172.21.15.6:0/411604108 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180035020 con 0x7f41900d2e50 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.652+0000 7f41949d0640 1 -- 172.21.15.6:0/411604108 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4180031830 con 0x7f41900d2e50 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.652+0000 7f41949d0640 1 -- 172.21.15.6:0/411604108 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180031c00 con 0x7f41900d2e50 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 -- 172.21.15.6:0/411604108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900d2e50 msgr2=0x7f41900b5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 --2- 172.21.15.6:0/411604108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900d2e50 0x7f41900b5740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f41900d16f0 tx=0x7f418002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 -- 172.21.15.6:0/411604108 shutdown_connections 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 --2- 172.21.15.6:0/411604108 >> v2:172.21.15.92:3300/0 conn(0x7f41900b5c80 0x7f41900b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 --2- 172.21.15.6:0/411604108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900d2e50 0x7f41900b5740 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 -- 172.21.15.6:0/411604108 >> 172.21.15.6:0/411604108 conn(0x7f4190072d10 msgr2=0x7f4190073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 -- 172.21.15.6:0/411604108 shutdown_connections 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.653+0000 7f4197c36640 1 -- 172.21.15.6:0/411604108 wait complete. 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.654+0000 7f4197c36640 1 Processor -- start 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.654+0000 7f4197c36640 1 -- start start 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900b5c80 0x7f4190155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41900d2e50 0x7f4190155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f419015dfd0 0x7f41901f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f41900b74b0 con 0x7f41900b5c80 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f41900b7330 con 0x7f419015dfd0 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f41900b7630 con 0x7f41900d2e50 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900b5c80 0x7f4190155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900b5c80 0x7f4190155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42864/0 (socket says 172.21.15.6:42864) 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 -- 172.21.15.6:0/661552579 learned_addr learned my addr 172.21.15.6:0/661552579 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 -- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41900d2e50 msgr2=0x7f4190155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41900d2e50 0x7f4190155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 -- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f419015dfd0 msgr2=0x7f41901f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41961d3640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f419015dfd0 0x7f41901f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f419015dfd0 0x7f41901f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 -- 172.21.15.6:0/661552579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4190153f60 con 0x7f41900b5c80 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41959d2640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900b5c80 0x7f4190155960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c460 tx=0x7f418c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41867fc640 1 -- 172.21.15.6:0/661552579 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c018070 con 0x7f41900b5c80 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41867fc640 1 -- 172.21.15.6:0/661552579 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418c007500 con 0x7f41900b5c80 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4190154240 con 0x7f41900b5c80 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f41867fc640 1 -- 172.21.15.6:0/661552579 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c004a20 con 0x7f41900b5c80 2024-03-18T14:41:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.655+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4190154800 con 0x7f41900b5c80 2024-03-18T14:41:34.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.656+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41900751b0 con 0x7f41900b5c80 2024-03-18T14:41:34.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.657+0000 7f41867fc640 1 -- 172.21.15.6:0/661552579 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f418c01b070 con 0x7f41900b5c80 2024-03-18T14:41:34.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.657+0000 7f41867fc640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f417406bbd0 0x7f417406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:34.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.657+0000 7f41867fc640 1 -- 172.21.15.6:0/661552579 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f418c089c70 con 0x7f41900b5c80 2024-03-18T14:41:34.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.657+0000 7f41951d1640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f417406bbd0 0x7f417406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:34.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.658+0000 7f41951d1640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f417406bbd0 0x7f417406e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4180036000 tx=0x7f4180030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:34.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.661+0000 7f41867fc640 1 -- 172.21.15.6:0/661552579 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f418c0587c0 con 0x7f41900b5c80 2024-03-18T14:41:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:34.881+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f41900cb160 con 0x7f41900b5c80 2024-03-18T14:41:34.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:41:34.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:41:35.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.534+0000 7f41867fc640 1 -- 172.21.15.6:0/661552579 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f418c0580c0 con 0x7f41900b5c80 2024-03-18T14:41:35.537 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T14:41:35.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.546+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f417406bbd0 msgr2=0x7f417406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:35.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.546+0000 7f4197c36640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f417406bbd0 0x7f417406e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4180036000 tx=0x7f4180030000 comp rx=0 tx=0).stop 2024-03-18T14:41:35.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.546+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900b5c80 msgr2=0x7f4190155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:35.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.546+0000 7f4197c36640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900b5c80 0x7f4190155960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c460 tx=0x7f418c00c920 comp rx=0 tx=0).stop 2024-03-18T14:41:35.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.547+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 shutdown_connections 2024-03-18T14:41:35.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.547+0000 7f4197c36640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f417406bbd0 0x7f417406e090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.547+0000 7f4197c36640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f419015dfd0 0x7f41901f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.547+0000 7f4197c36640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41900d2e50 0x7f4190155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.547+0000 7f4197c36640 1 --2- 172.21.15.6:0/661552579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41900b5c80 0x7f4190155960 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c460 tx=0x7f418c00c920 comp rx=0 tx=0).stop 2024-03-18T14:41:35.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.547+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 >> 172.21.15.6:0/661552579 conn(0x7f4190072d10 msgr2=0x7f41900cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:35.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.549+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 shutdown_connections 2024-03-18T14:41:35.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.550+0000 7f4197c36640 1 -- 172.21.15.6:0/661552579 wait complete. 2024-03-18T14:41:35.573 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T14:41:35.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T14:41:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.682+0000 7f01ee11d640 1 Processor -- start 2024-03-18T14:41:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.682+0000 7f01ee11d640 1 -- start start 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01ee11d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01e815e0b0 0x7f01e80b95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01ee11d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f01e806c730 0x7f01e806cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01ee11d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80d22c0 con 0x7f01e80d1de0 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01ee11d640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f01e806d0a0 con 0x7f01e815e0b0 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01ee11d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f01e80d25a0 con 0x7f01e806c730 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e77fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01e815e0b0 0x7f01e80b95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e77fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01e815e0b0 0x7f01e80b95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42872/0 (socket says 172.21.15.6:42872) 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e77fe640 1 -- 172.21.15.6:0/2554889934 learned_addr learned my addr 172.21.15.6:0/2554889934 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e7fff640 1 --2- 172.21.15.6:0/2554889934 >> v2:172.21.15.92:3300/0 conn(0x7f01e806c730 0x7f01e806cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e77fe640 1 -- 172.21.15.6:0/2554889934 >> v2:172.21.15.92:3300/0 conn(0x7f01e806c730 msgr2=0x7f01e806cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e77fe640 1 --2- 172.21.15.6:0/2554889934 >> v2:172.21.15.92:3300/0 conn(0x7f01e806c730 0x7f01e806cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e77fe640 1 -- 172.21.15.6:0/2554889934 >> v1:172.21.15.92:6789/0 conn(0x7f01e80d1de0 legacy=0x7f01e80d3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e77fe640 1 -- 172.21.15.6:0/2554889934 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e80d2880 con 0x7f01e815e0b0 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.683+0000 7f01e7fff640 1 --2- 172.21.15.6:0/2554889934 >> v2:172.21.15.92:3300/0 conn(0x7f01e806c730 0x7f01e806cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.684+0000 7f01e77fe640 1 --2- 172.21.15.6:0/2554889934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e815e0b0 0x7f01e80b95f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f01e80d1250 tx=0x7f01d802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f517d5bc0b4b25f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.684+0000 7f01e67fc640 1 -- 172.21.15.6:0/2554889934 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d802f850 con 0x7f01e815e0b0 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.684+0000 7f01e67fc640 1 -- 172.21.15.6:0/2554889934 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d802f9f0 con 0x7f01e815e0b0 2024-03-18T14:41:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.684+0000 7f01e67fc640 1 -- 172.21.15.6:0/2554889934 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d802fdc0 con 0x7f01e815e0b0 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 -- 172.21.15.6:0/2554889934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e815e0b0 msgr2=0x7f01e80b95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2554889934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e815e0b0 0x7f01e80b95f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f01e80d1250 tx=0x7f01d802ba30 comp rx=0 tx=0).stop 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 -- 172.21.15.6:0/2554889934 shutdown_connections 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2554889934 >> v2:172.21.15.92:3300/0 conn(0x7f01e806c730 0x7f01e806cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2554889934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e815e0b0 0x7f01e80b95f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 -- 172.21.15.6:0/2554889934 >> 172.21.15.6:0/2554889934 conn(0x7f01e8072ce0 msgr2=0x7f01e80730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 -- 172.21.15.6:0/2554889934 shutdown_connections 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.685+0000 7f01ee11d640 1 -- 172.21.15.6:0/2554889934 wait complete. 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.686+0000 7f01ee11d640 1 Processor -- start 2024-03-18T14:41:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.686+0000 7f01ee11d640 1 -- start start 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01ee11d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e806c730 0x7f01e815e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01ee11d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01e80d1de0 0x7f01e80cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01ee11d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e815e0b0 0x7f01e80cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01ee11d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f01e80b9b30 con 0x7f01e806c730 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01ee11d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f01e80d2e40 con 0x7f01e815e0b0 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01ee11d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f01e81f5520 con 0x7f01e80d1de0 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e806c730 0x7f01e815e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e806c730 0x7f01e815e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42876/0 (socket says 172.21.15.6:42876) 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 -- 172.21.15.6:0/2570245774 learned_addr learned my addr 172.21.15.6:0/2570245774 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e6ffd640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01e80d1de0 0x7f01e80cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e7fff640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e815e0b0 0x7f01e80cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 -- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01e80d1de0 msgr2=0x7f01e80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01e80d1de0 0x7f01e80cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 -- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e815e0b0 msgr2=0x7f01e80cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e815e0b0 0x7f01e80cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.687+0000 7f01e77fe640 1 -- 172.21.15.6:0/2570245774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e815ea60 con 0x7f01e806c730 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.688+0000 7f01e77fe640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e806c730 0x7f01e815e520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01d802ba70 tx=0x7f01d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.688+0000 7f01cffff640 1 -- 172.21.15.6:0/2570245774 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8037c90 con 0x7f01e806c730 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.688+0000 7f01cffff640 1 -- 172.21.15.6:0/2570245774 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d8046070 con 0x7f01e806c730 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.688+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e81f49a0 con 0x7f01e806c730 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.688+0000 7f01cffff640 1 -- 172.21.15.6:0/2570245774 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8041640 con 0x7f01e806c730 2024-03-18T14:41:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.688+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e81f4f60 con 0x7f01e806c730 2024-03-18T14:41:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.689+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e808afe0 con 0x7f01e806c730 2024-03-18T14:41:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.689+0000 7f01cffff640 1 -- 172.21.15.6:0/2570245774 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f01d8040070 con 0x7f01e806c730 2024-03-18T14:41:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.690+0000 7f01cffff640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f01d406bbd0 0x7f01d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.690+0000 7f01cffff640 1 -- 172.21.15.6:0/2570245774 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f01d80ad050 con 0x7f01e806c730 2024-03-18T14:41:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.690+0000 7f01e6ffd640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f01d406bbd0 0x7f01d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.690+0000 7f01e6ffd640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f01d406bbd0 0x7f01d406e090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f01e8161590 tx=0x7f01dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.694+0000 7f01cffff640 1 -- 172.21.15.6:0/2570245774 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f01d8033030 con 0x7f01e806c730 2024-03-18T14:41:35.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:35.884+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f01e804f620 con 0x7f01e806c730 2024-03-18T14:41:36.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.535+0000 7f01cffff640 1 -- 172.21.15.6:0/2570245774 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f01d807b500 con 0x7f01e806c730 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.538+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f01d406bbd0 msgr2=0x7f01d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.538+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f01d406bbd0 0x7f01d406e090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f01e8161590 tx=0x7f01dc009040 comp rx=0 tx=0).stop 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.538+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e806c730 msgr2=0x7f01e815e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.538+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e806c730 0x7f01e815e520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01d802ba70 tx=0x7f01d8004730 comp rx=0 tx=0).stop 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 shutdown_connections 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f01d406bbd0 0x7f01d406e090 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e815e0b0 0x7f01e80cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01e80d1de0 0x7f01e80cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 --2- 172.21.15.6:0/2570245774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01e806c730 0x7f01e815e520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 >> 172.21.15.6:0/2570245774 conn(0x7f01e8072ce0 msgr2=0x7f01e80c7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 shutdown_connections 2024-03-18T14:41:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.539+0000 7f01ee11d640 1 -- 172.21.15.6:0/2570245774 wait complete. 2024-03-18T14:41:36.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T14:41:36.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.670+0000 7fdda2e30640 1 Processor -- start 2024-03-18T14:41:36.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.670+0000 7fdda2e30640 1 -- start start 2024-03-18T14:41:36.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.670+0000 7fdda2e30640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdd9c06c7a0 0x7fdd9c0d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:36.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.670+0000 7fdda2e30640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd9c0b6df0 0x7fdd9c0b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:36.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.670+0000 7fdda2e30640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd9c0d1f30 con 0x7fdd9c06cb80 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.670+0000 7fdda2e30640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fdd9c0b9cf0 con 0x7fdd9c0b6df0 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.670+0000 7fdda2e30640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdd9c0d2210 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda13cd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd9c0b6df0 0x7fdd9c0b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda13cd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd9c0b6df0 0x7fdd9c0b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42886/0 (socket says 172.21.15.6:42886) 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda13cd640 1 -- 172.21.15.6:0/2421713975 learned_addr learned my addr 172.21.15.6:0/2421713975 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda0bcc640 1 --2- 172.21.15.6:0/2421713975 >> v2:172.21.15.92:3300/0 conn(0x7fdd9c06c7a0 0x7fdd9c0d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda13cd640 1 -- 172.21.15.6:0/2421713975 >> v2:172.21.15.92:3300/0 conn(0x7fdd9c06c7a0 msgr2=0x7fdd9c0d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda13cd640 1 --2- 172.21.15.6:0/2421713975 >> v2:172.21.15.92:3300/0 conn(0x7fdd9c06c7a0 0x7fdd9c0d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda13cd640 1 -- 172.21.15.6:0/2421713975 >> v1:172.21.15.92:6789/0 conn(0x7fdd9c06cb80 legacy=0x7fdd9c0b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdd9b7fe640 1 -- 172.21.15.6:0/2421713975 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2521437100 0 0) 0x7fdd9c0d1f30 con 0x7fdd9c06cb80 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.671+0000 7fdda13cd640 1 -- 172.21.15.6:0/2421713975 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c0d24f0 con 0x7fdd9c0b6df0 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdda13cd640 1 --2- 172.21.15.6:0/2421713975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c0b7280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0d0fa0 tx=0x7fdd9402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221dd19a9ee14c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdd9b7fe640 1 -- 172.21.15.6:0/2421713975 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9402cbb0 con 0x7fdd9c0b6df0 2024-03-18T14:41:36.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdd9b7fe640 1 -- 172.21.15.6:0/2421713975 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9402cd50 con 0x7fdd9c0b6df0 2024-03-18T14:41:36.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdd9b7fe640 1 -- 172.21.15.6:0/2421713975 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94034610 con 0x7fdd9c0b6df0 2024-03-18T14:41:36.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdda2e30640 1 -- 172.21.15.6:0/2421713975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c0b6df0 msgr2=0x7fdd9c0b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2421713975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c0b7280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0d0fa0 tx=0x7fdd9402c010 comp rx=0 tx=0).stop 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdda2e30640 1 -- 172.21.15.6:0/2421713975 shutdown_connections 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2421713975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c0b7280 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.672+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2421713975 >> v2:172.21.15.92:3300/0 conn(0x7fdd9c06c7a0 0x7fdd9c0d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.673+0000 7fdda2e30640 1 -- 172.21.15.6:0/2421713975 >> 172.21.15.6:0/2421713975 conn(0x7fdd9c072ce0 msgr2=0x7fdd9c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.673+0000 7fdda2e30640 1 -- 172.21.15.6:0/2421713975 shutdown_connections 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.673+0000 7fdda2e30640 1 -- 172.21.15.6:0/2421713975 wait complete. 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.674+0000 7fdda2e30640 1 Processor -- start 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.674+0000 7fdda2e30640 1 -- start start 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.674+0000 7fdda2e30640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c06c7a0 0x7fdd9c14f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda2e30640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd9c06cb80 0x7fdd9c1f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:36.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda2e30640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c1f7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda2e30640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c06cfe0 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda2e30640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c0d2ab0 con 0x7fdd9c0b6df0 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda2e30640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9c0d37f0 con 0x7fdd9c06cb80 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c06c7a0 0x7fdd9c14f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c06c7a0 0x7fdd9c14f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42888/0 (socket says 172.21.15.6:42888) 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 -- 172.21.15.6:0/2265278257 learned_addr learned my addr 172.21.15.6:0/2265278257 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda13cd640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c1f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdd9bfff640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd9c06cb80 0x7fdd9c1f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 -- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd9c06cb80 msgr2=0x7fdd9c1f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd9c06cb80 0x7fdd9c1f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 -- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd9c0b6df0 msgr2=0x7fdd9c1f7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c1f7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 -- 172.21.15.6:0/2265278257 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c1f47f0 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda13cd640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c1f7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda0bcc640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c06c7a0 0x7fdd9c14f3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e9b0 tx=0x7fdd9000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdd997fa640 1 -- 172.21.15.6:0/2265278257 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90017070 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdd997fa640 1 -- 172.21.15.6:0/2265278257 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9000c9a0 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9c1f4ad0 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdd997fa640 1 -- 172.21.15.6:0/2265278257 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9000cd90 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.675+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd9c1f5090 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.677+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd9c04f5b0 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.677+0000 7fdd997fa640 1 -- 172.21.15.6:0/2265278257 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdd9001a070 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.677+0000 7fdd997fa640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8006bbd0 0x7fdd8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.678+0000 7fdd997fa640 1 -- 172.21.15.6:0/2265278257 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7fdd9008ab60 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.678+0000 7fdd9bfff640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8006bbd0 0x7fdd8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:36.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.678+0000 7fdd9bfff640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8006bbd0 0x7fdd8006e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c1530f0 tx=0x7fdd88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:36.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.682+0000 7fdd997fa640 1 -- 172.21.15.6:0/2265278257 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdd90058ed0 con 0x7fdd9c06c7a0 2024-03-18T14:41:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:36.866+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fdd9c15d840 con 0x7fdd9c06c7a0 2024-03-18T14:41:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.550+0000 7fdd997fa640 1 -- 172.21.15.6:0/2265278257 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7fdd9005cf70 con 0x7fdd9c06c7a0 2024-03-18T14:41:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data_ec' created 2024-03-18T14:41:38.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.555+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8006bbd0 msgr2=0x7fdd8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.555+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8006bbd0 0x7fdd8006e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c1530f0 tx=0x7fdd88009040 comp rx=0 tx=0).stop 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.555+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c06c7a0 msgr2=0x7fdd9c14f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.555+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c06c7a0 0x7fdd9c14f3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e9b0 tx=0x7fdd9000ee70 comp rx=0 tx=0).stop 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.556+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 shutdown_connections 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.556+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8006bbd0 0x7fdd8006e090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.556+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd9c0b6df0 0x7fdd9c1f7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.556+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd9c06cb80 0x7fdd9c1f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.556+0000 7fdda2e30640 1 --2- 172.21.15.6:0/2265278257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd9c06c7a0 0x7fdd9c14f3e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.556+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 >> 172.21.15.6:0/2265278257 conn(0x7fdd9c072ce0 msgr2=0x7fdd9c0cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:38.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.561+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 shutdown_connections 2024-03-18T14:41:38.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.562+0000 7fdda2e30640 1 -- 172.21.15.6:0/2265278257 wait complete. 2024-03-18T14:41:38.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T14:41:38.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.687+0000 7f187a461640 1 Processor -- start 2024-03-18T14:41:38.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.687+0000 7f187a461640 1 -- start start 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f187a461640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f187415bc00 0x7f18740d1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f187a461640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18740d0040 0x7f18740d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f187a461640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18740d06a0 con 0x7f187415bfe0 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f187a461640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f18740ccd70 con 0x7f187415bc00 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f187a461640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f18740ce210 con 0x7f18740d0040 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f18737fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f187415bfe0 0x7f18740cf930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42918/0 (socket says 172.21.15.6:42918) 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f18737fe640 1 -- 172.21.15.6:0/2373605918 learned_addr learned my addr 172.21.15.6:0/2373605918 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f18789fe640 1 --2- 172.21.15.6:0/2373605918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18740d0040 0x7f18740d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f1873fff640 1 --2- 172.21.15.6:0/2373605918 >> v2:172.21.15.92:3300/0 conn(0x7f187415bc00 0x7f18740d1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f1872ffd640 1 -- 172.21.15.6:0/2373605918 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1878962109 0 0) 0x7f18740d06a0 con 0x7f187415bfe0 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f1872ffd640 1 -- 172.21.15.6:0/2373605918 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f185c003520 con 0x7f187415bfe0 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f18789fe640 1 -- 172.21.15.6:0/2373605918 >> v2:172.21.15.92:3300/0 conn(0x7f187415bc00 msgr2=0x7f18740d1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f18789fe640 1 --2- 172.21.15.6:0/2373605918 >> v2:172.21.15.92:3300/0 conn(0x7f187415bc00 0x7f18740d1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f18789fe640 1 -- 172.21.15.6:0/2373605918 >> v1:172.21.15.6:6789/0 conn(0x7f187415bfe0 legacy=0x7f18740cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f18789fe640 1 -- 172.21.15.6:0/2373605918 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18740d0980 con 0x7f18740d0040 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.688+0000 7f1873fff640 1 --2- 172.21.15.6:0/2373605918 >> v2:172.21.15.92:3300/0 conn(0x7f187415bc00 0x7f18740d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.689+0000 7f18789fe640 1 --2- 172.21.15.6:0/2373605918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18740d0040 0x7f18740d1dc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f186c009d90 tx=0x7f186c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=375f4381c115c5ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.689+0000 7f1872ffd640 1 -- 172.21.15.6:0/2373605918 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c039070 con 0x7f18740d0040 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.689+0000 7f1872ffd640 1 -- 172.21.15.6:0/2373605918 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f186c02cc50 con 0x7f18740d0040 2024-03-18T14:41:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.689+0000 7f1872ffd640 1 -- 172.21.15.6:0/2373605918 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c034440 con 0x7f18740d0040 2024-03-18T14:41:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 -- 172.21.15.6:0/2373605918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18740d0040 msgr2=0x7f18740d1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 --2- 172.21.15.6:0/2373605918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18740d0040 0x7f18740d1dc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f186c009d90 tx=0x7f186c02bea0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 -- 172.21.15.6:0/2373605918 shutdown_connections 2024-03-18T14:41:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 --2- 172.21.15.6:0/2373605918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18740d0040 0x7f18740d1dc0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 --2- 172.21.15.6:0/2373605918 >> v2:172.21.15.92:3300/0 conn(0x7f187415bc00 0x7f18740d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 -- 172.21.15.6:0/2373605918 >> 172.21.15.6:0/2373605918 conn(0x7f187409ae00 msgr2=0x7f187408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:38.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 -- 172.21.15.6:0/2373605918 shutdown_connections 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.690+0000 7f187a461640 1 -- 172.21.15.6:0/2373605918 wait complete. 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.691+0000 7f187a461640 1 Processor -- start 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.691+0000 7f187a461640 1 -- start start 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f187a461640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18740d0040 0x7f18740b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f187a461640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187415bc00 0x7f18740b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f187a461640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f187415bfe0 0x7f18740ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f187a461640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18740ae6f0 con 0x7f187415bc00 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f187a461640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f18740cc6b0 con 0x7f18740d0040 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f187a461640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f18740cd200 con 0x7f187415bfe0 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187415bc00 0x7f18740b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187415bc00 0x7f18740b70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42904/0 (socket says 172.21.15.6:42904) 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 -- 172.21.15.6:0/1229437393 learned_addr learned my addr 172.21.15.6:0/1229437393 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18789fe640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f187415bfe0 0x7f18740ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f1873fff640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18740d0040 0x7f18740b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 -- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f187415bfe0 msgr2=0x7f18740ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f187415bfe0 0x7f18740ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 -- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18740d0040 msgr2=0x7f18740b6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18740d0040 0x7f18740b6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 -- 172.21.15.6:0/1229437393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1874156ae0 con 0x7f187415bc00 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.692+0000 7f18737fe640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187415bc00 0x7f18740b70f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f186400c390 tx=0x7f186400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.693+0000 7f1870ff9640 1 -- 172.21.15.6:0/1229437393 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864017070 con 0x7f187415bc00 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.693+0000 7f1870ff9640 1 -- 172.21.15.6:0/1229437393 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1864005ce0 con 0x7f187415bc00 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.693+0000 7f1870ff9640 1 -- 172.21.15.6:0/1229437393 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18640072f0 con 0x7f187415bc00 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.693+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1874156d80 con 0x7f187415bc00 2024-03-18T14:41:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.693+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18740b7630 con 0x7f187415bc00 2024-03-18T14:41:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.694+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1874157170 con 0x7f187415bc00 2024-03-18T14:41:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.694+0000 7f1870ff9640 1 -- 172.21.15.6:0/1229437393 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f186401a070 con 0x7f187415bc00 2024-03-18T14:41:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.694+0000 7f1870ff9640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f186006bbd0 0x7f186006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.695+0000 7f1870ff9640 1 -- 172.21.15.6:0/1229437393 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f186408a110 con 0x7f187415bc00 2024-03-18T14:41:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.695+0000 7f1873fff640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f186006bbd0 0x7f186006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.695+0000 7f1873fff640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f186006bbd0 0x7f186006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1868009560 tx=0x7f1868009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:38.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.699+0000 7f1870ff9640 1 -- 172.21.15.6:0/1229437393 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1864058290 con 0x7f187415bc00 2024-03-18T14:41:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:38.881+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f1874156f50 con 0x7f187415bc00 2024-03-18T14:41:39.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.554+0000 7f1870ff9640 1 -- 172.21.15.6:0/1229437393 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7f186405c330 con 0x7f187415bc00 2024-03-18T14:41:39.557 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T14:41:39.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.558+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f186006bbd0 msgr2=0x7f186006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:39.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.558+0000 7f187a461640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f186006bbd0 0x7f186006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1868009560 tx=0x7f1868009190 comp rx=0 tx=0).stop 2024-03-18T14:41:39.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.558+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187415bc00 msgr2=0x7f18740b70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:39.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187415bc00 0x7f18740b70f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f186400c390 tx=0x7f186400c850 comp rx=0 tx=0).stop 2024-03-18T14:41:39.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 shutdown_connections 2024-03-18T14:41:39.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f186006bbd0 0x7f186006e090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f187415bfe0 0x7f18740ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187415bc00 0x7f18740b70f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 --2- 172.21.15.6:0/1229437393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18740d0040 0x7f18740b6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 >> 172.21.15.6:0/1229437393 conn(0x7f187409ae00 msgr2=0x7f1874070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:39.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 shutdown_connections 2024-03-18T14:41:39.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.559+0000 7f187a461640 1 -- 172.21.15.6:0/1229437393 wait complete. 2024-03-18T14:41:39.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T14:41:39.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.693+0000 7f78cf843640 1 Processor -- start 2024-03-18T14:41:39.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cf843640 1 -- start start 2024-03-18T14:41:39.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cf843640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f78c806b810 0x7f78c80ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cf843640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78c815d640 0x7f78c815daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cf843640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c80b22c0 con 0x7f78c806b430 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cf843640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f78c815e650 con 0x7f78c815d640 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cf843640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f78c815e930 con 0x7f78c806b810 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cdde0640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78c815d640 0x7f78c815daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cd5df640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f78c806b430 0x7f78c80b1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42926/0 (socket says 172.21.15.6:42926) 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.694+0000 7f78cd5df640 1 -- 172.21.15.6:0/3871434549 learned_addr learned my addr 172.21.15.6:0/3871434549 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.695+0000 7f78ccdde640 1 --2- 172.21.15.6:0/3871434549 >> v2:172.21.15.92:3300/0 conn(0x7f78c806b810 0x7f78c80ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.695+0000 7f78cdde0640 1 -- 172.21.15.6:0/3871434549 >> v2:172.21.15.92:3300/0 conn(0x7f78c806b810 msgr2=0x7f78c80ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.695+0000 7f78cdde0640 1 --2- 172.21.15.6:0/3871434549 >> v2:172.21.15.92:3300/0 conn(0x7f78c806b810 0x7f78c80ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.695+0000 7f78cdde0640 1 -- 172.21.15.6:0/3871434549 >> v1:172.21.15.6:6789/0 conn(0x7f78c806b430 legacy=0x7f78c80b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.695+0000 7f78c7fff640 1 -- 172.21.15.6:0/3871434549 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3827891297 0 0) 0x7f78c80b22c0 con 0x7f78c806b430 2024-03-18T14:41:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.695+0000 7f78cdde0640 1 -- 172.21.15.6:0/3871434549 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c815ec10 con 0x7f78c815d640 2024-03-18T14:41:39.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.695+0000 7f78cdde0640 1 --2- 172.21.15.6:0/3871434549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c815daf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78c806ddd0 tx=0x7f78c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34967f9436ca97ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:39.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78c7fff640 1 -- 172.21.15.6:0/3871434549 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0035020 con 0x7f78c815d640 2024-03-18T14:41:39.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78c7fff640 1 -- 172.21.15.6:0/3871434549 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78c0031830 con 0x7f78c815d640 2024-03-18T14:41:39.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78c7fff640 1 -- 172.21.15.6:0/3871434549 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0031c00 con 0x7f78c815d640 2024-03-18T14:41:39.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 -- 172.21.15.6:0/3871434549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 msgr2=0x7f78c815daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:39.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 --2- 172.21.15.6:0/3871434549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c815daf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78c806ddd0 tx=0x7f78c002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 -- 172.21.15.6:0/3871434549 shutdown_connections 2024-03-18T14:41:39.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 --2- 172.21.15.6:0/3871434549 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c815daf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 --2- 172.21.15.6:0/3871434549 >> v2:172.21.15.92:3300/0 conn(0x7f78c806b810 0x7f78c80ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 -- 172.21.15.6:0/3871434549 >> 172.21.15.6:0/3871434549 conn(0x7f78c8099a50 msgr2=0x7f78c80d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:39.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 -- 172.21.15.6:0/3871434549 shutdown_connections 2024-03-18T14:41:39.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.696+0000 7f78cf843640 1 -- 172.21.15.6:0/3871434549 wait complete. 2024-03-18T14:41:39.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.697+0000 7f78cf843640 1 Processor -- start 2024-03-18T14:41:39.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cf843640 1 -- start start 2024-03-18T14:41:39.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cf843640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78c806b430 0x7f78c80b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:39.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cf843640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c806b810 0x7f78c8160770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:39.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cf843640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c8160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:39.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cf843640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f78c80cce70 con 0x7f78c815d640 2024-03-18T14:41:39.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cf843640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f78c80cccf0 con 0x7f78c806b810 2024-03-18T14:41:39.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cf843640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f78c815e030 con 0x7f78c806b430 2024-03-18T14:41:39.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cdde0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c8160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cdde0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c8160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42920/0 (socket says 172.21.15.6:42920) 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cdde0640 1 -- 172.21.15.6:0/2079798137 learned_addr learned my addr 172.21.15.6:0/2079798137 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78cd5df640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78c806b430 0x7f78c80b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.698+0000 7f78ccdde640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c806b810 0x7f78c8160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cdde0640 1 -- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78c806b430 msgr2=0x7f78c80b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cdde0640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78c806b430 0x7f78c80b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cdde0640 1 -- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c806b810 msgr2=0x7f78c8160770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cdde0640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c806b810 0x7f78c8160770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cdde0640 1 -- 172.21.15.6:0/2079798137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c814fcd0 con 0x7f78c815d640 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78ccdde640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c806b810 0x7f78c8160770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cdde0640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c8160cb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78c002b910 tx=0x7f78c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78c5ffb640 1 -- 172.21.15.6:0/2079798137 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c003a8b0 con 0x7f78c815d640 2024-03-18T14:41:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78c5ffb640 1 -- 172.21.15.6:0/2079798137 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78c003aa70 con 0x7f78c815d640 2024-03-18T14:41:39.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c814ffb0 con 0x7f78c815d640 2024-03-18T14:41:39.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78c5ffb640 1 -- 172.21.15.6:0/2079798137 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c003ae60 con 0x7f78c815d640 2024-03-18T14:41:39.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.699+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c80c4520 con 0x7f78c815d640 2024-03-18T14:41:39.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.700+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c80751b0 con 0x7f78c815d640 2024-03-18T14:41:39.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.700+0000 7f78c5ffb640 1 -- 172.21.15.6:0/2079798137 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f78c0038020 con 0x7f78c815d640 2024-03-18T14:41:39.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.701+0000 7f78c5ffb640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f78b406bc00 0x7f78b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:39.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.701+0000 7f78c5ffb640 1 -- 172.21.15.6:0/2079798137 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f78c00ac870 con 0x7f78c815d640 2024-03-18T14:41:39.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.701+0000 7f78cd5df640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f78b406bc00 0x7f78b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:39.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.701+0000 7f78cd5df640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f78b406bc00 0x7f78b406e0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78b80012e0 tx=0x7f78b8001700 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.705+0000 7f78c5ffb640 1 -- 172.21.15.6:0/2079798137 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f78c003f070 con 0x7f78c815d640 2024-03-18T14:41:39.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:39.928+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f78c80d1380 con 0x7f78c815d640 2024-03-18T14:41:39.956 INFO:tasks.ceph.mds.d.smithi092.stderr:2024-03-18T14:41:39.954+0000 7fb870ff2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T14:41:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.567+0000 7f78c5ffb640 1 -- 172.21.15.6:0/2079798137 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7f78c007aa80 con 0x7f78c815d640 2024-03-18T14:41:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:added data pool 4 to fsmap 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f78b406bc00 msgr2=0x7f78b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f78b406bc00 0x7f78b406e0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78b80012e0 tx=0x7f78b8001700 comp rx=0 tx=0).stop 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 msgr2=0x7f78c8160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c8160cb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78c002b910 tx=0x7f78c00046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 shutdown_connections 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f78b406bc00 0x7f78b406e0c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78c815d640 0x7f78c8160cb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c806b810 0x7f78c8160770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 --2- 172.21.15.6:0/2079798137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78c806b430 0x7f78c80b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 >> 172.21.15.6:0/2079798137 conn(0x7f78c8099a50 msgr2=0x7f78c806f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 shutdown_connections 2024-03-18T14:41:40.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.570+0000 7f78cf843640 1 -- 172.21.15.6:0/2079798137 wait complete. 2024-03-18T14:41:40.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:41:40.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.690+0000 7fa7159e6640 1 Processor -- start 2024-03-18T14:41:40.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.690+0000 7fa7159e6640 1 -- start start 2024-03-18T14:41:40.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa7159e6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa7100ced00 0x7fa710162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:40.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa7159e6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7100d3210 0x7fa7100d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:40.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa7159e6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7100b9660 con 0x7fa71015e020 2024-03-18T14:41:40.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa7159e6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa7100d2270 con 0x7fa7100d3210 2024-03-18T14:41:40.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa7159e6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa7100ce390 con 0x7fa7100ced00 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7100d3210 0x7fa7100d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7100d3210 0x7fa7100d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42932/0 (socket says 172.21.15.6:42932) 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70f7fe640 1 -- 172.21.15.6:0/1465128118 learned_addr learned my addr 172.21.15.6:0/1465128118 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70e7fc640 1 --2- 172.21.15.6:0/1465128118 >> v2:172.21.15.92:3300/0 conn(0x7fa7100ced00 0x7fa710162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70f7fe640 1 -- 172.21.15.6:0/1465128118 >> v2:172.21.15.92:3300/0 conn(0x7fa7100ced00 msgr2=0x7fa710162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70f7fe640 1 --2- 172.21.15.6:0/1465128118 >> v2:172.21.15.92:3300/0 conn(0x7fa7100ced00 0x7fa710162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70f7fe640 1 -- 172.21.15.6:0/1465128118 >> v1:172.21.15.92:6789/0 conn(0x7fa71015e020 legacy=0x7fa7100d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.691+0000 7fa70f7fe640 1 -- 172.21.15.6:0/1465128118 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7100b9900 con 0x7fa7100d3210 2024-03-18T14:41:40.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.692+0000 7fa70f7fe640 1 --2- 172.21.15.6:0/1465128118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7100d3210 0x7fa7100d3c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7100d4da0 tx=0x7fa70802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac51e491b7dcea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.692+0000 7fa70dffb640 1 -- 172.21.15.6:0/1465128118 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708035020 con 0x7fa7100d3210 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.692+0000 7fa70dffb640 1 -- 172.21.15.6:0/1465128118 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa708031870 con 0x7fa7100d3210 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.692+0000 7fa70dffb640 1 -- 172.21.15.6:0/1465128118 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708031c40 con 0x7fa7100d3210 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 -- 172.21.15.6:0/1465128118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7100d3210 msgr2=0x7fa7100d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 --2- 172.21.15.6:0/1465128118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7100d3210 0x7fa7100d3c30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7100d4da0 tx=0x7fa70802b910 comp rx=0 tx=0).stop 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 -- 172.21.15.6:0/1465128118 shutdown_connections 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 --2- 172.21.15.6:0/1465128118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7100d3210 0x7fa7100d3c30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 --2- 172.21.15.6:0/1465128118 >> v2:172.21.15.92:3300/0 conn(0x7fa7100ced00 0x7fa710162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 -- 172.21.15.6:0/1465128118 >> 172.21.15.6:0/1465128118 conn(0x7fa710072ce0 msgr2=0x7fa7100730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 -- 172.21.15.6:0/1465128118 shutdown_connections 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.693+0000 7fa7159e6640 1 -- 172.21.15.6:0/1465128118 wait complete. 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.694+0000 7fa7159e6640 1 Processor -- start 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.694+0000 7fa7159e6640 1 -- start start 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa7159e6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7100ced00 0x7fa7101540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa7159e6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7100d3210 0x7fa7101e8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa7159e6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa71015e020 0x7fa71015f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa7159e6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7100cf2c0 con 0x7fa71015e020 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa7159e6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa7100ce890 con 0x7fa7100d3210 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa7159e6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa7100cf440 con 0x7fa7100ced00 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa71015e020 0x7fa71015f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:40.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa71015e020 0x7fa71015f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42938/0 (socket says 172.21.15.6:42938) 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 -- 172.21.15.6:0/4038055347 learned_addr learned my addr 172.21.15.6:0/4038055347 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70e7fc640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7100d3210 0x7fa7101e8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70effd640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7100ced00 0x7fa7101540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 -- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7100ced00 msgr2=0x7fa7101540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7100ced00 0x7fa7101540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 -- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7100d3210 msgr2=0x7fa7101e8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7100d3210 0x7fa7101e8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 -- 172.21.15.6:0/4038055347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7100b64c0 con 0x7fa71015e020 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70e7fc640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7100d3210 0x7fa7101e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa70f7fe640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa71015e020 0x7fa71015f970 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa70802b950 tx=0x7fa7080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.695+0000 7fa707fff640 1 -- 172.21.15.6:0/4038055347 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708040070 con 0x7fa71015e020 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.696+0000 7fa707fff640 1 -- 172.21.15.6:0/4038055347 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa708030d00 con 0x7fa71015e020 2024-03-18T14:41:40.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.696+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7101f8ad0 con 0x7fa71015e020 2024-03-18T14:41:40.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.696+0000 7fa707fff640 1 -- 172.21.15.6:0/4038055347 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708035040 con 0x7fa71015e020 2024-03-18T14:41:40.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.696+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7101f9090 con 0x7fa71015e020 2024-03-18T14:41:40.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.697+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa71008afe0 con 0x7fa71015e020 2024-03-18T14:41:40.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.697+0000 7fa707fff640 1 -- 172.21.15.6:0/4038055347 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa708038020 con 0x7fa71015e020 2024-03-18T14:41:40.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.697+0000 7fa707fff640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6f006bbd0 0x7fa6f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:40.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.698+0000 7fa707fff640 1 -- 172.21.15.6:0/4038055347 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa7080aca50 con 0x7fa71015e020 2024-03-18T14:41:40.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.698+0000 7fa70effd640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6f006bbd0 0x7fa6f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:40.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.698+0000 7fa70effd640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6f006bbd0 0x7fa6f006e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7000012e0 tx=0x7fa700009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:40.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.702+0000 7fa707fff640 1 -- 172.21.15.6:0/4038055347 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa70807ac40 con 0x7fa71015e020 2024-03-18T14:41:40.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.929+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa71004f620 con 0x7fa71015e020 2024-03-18T14:41:40.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.930+0000 7fa707fff640 1 -- 172.21.15.6:0/4038055347 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5168 (secure 0 0 0) 0x7fa70807ece0 con 0x7fa71015e020 2024-03-18T14:41:40.931 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:40.931 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:39.929725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:41:40.932 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 6 2024-03-18T14:41:40.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6f006bbd0 msgr2=0x7fa6f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6f006bbd0 0x7fa6f006e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7000012e0 tx=0x7fa700009040 comp rx=0 tx=0).stop 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa71015e020 msgr2=0x7fa71015f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa71015e020 0x7fa71015f970 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa70802b950 tx=0x7fa7080046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 shutdown_connections 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6f006bbd0 0x7fa6f006e090 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa71015e020 0x7fa71015f970 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7100d3210 0x7fa7101e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 --2- 172.21.15.6:0/4038055347 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7100ced00 0x7fa7101540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.931+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 >> 172.21.15.6:0/4038055347 conn(0x7fa710072ce0 msgr2=0x7fa7100b8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.932+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 shutdown_connections 2024-03-18T14:41:40.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:40.932+0000 7fa7159e6640 1 -- 172.21.15.6:0/4038055347 wait complete. 2024-03-18T14:41:40.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:41.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.044+0000 7f8f71e97640 1 Processor -- start 2024-03-18T14:41:41.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.044+0000 7f8f71e97640 1 -- start start 2024-03-18T14:41:41.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f71e97640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f6c0ced00 0x7f8f6c162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f71e97640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c0d3210 0x7f8f6c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f71e97640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f6c0b9660 con 0x7f8f6c15e020 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f71e97640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8f6c0d2270 con 0x7f8f6c0ced00 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f71e97640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6c0ce390 con 0x7f8f6c0d3210 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6affd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f6c0ced00 0x7f8f6c162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6affd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f6c0ced00 0x7f8f6c162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42946/0 (socket says 172.21.15.6:42946) 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6affd640 1 -- 172.21.15.6:0/2315190267 learned_addr learned my addr 172.21.15.6:0/2315190267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6bfff640 1 --2- 172.21.15.6:0/2315190267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c0d3210 0x7f8f6c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6affd640 1 -- 172.21.15.6:0/2315190267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c0d3210 msgr2=0x7f8f6c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6affd640 1 --2- 172.21.15.6:0/2315190267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c0d3210 0x7f8f6c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6affd640 1 -- 172.21.15.6:0/2315190267 >> v1:172.21.15.92:6789/0 conn(0x7f8f6c15e020 legacy=0x7f8f6c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.045+0000 7f8f6affd640 1 -- 172.21.15.6:0/2315190267 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c0b9900 con 0x7f8f6c0ced00 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f6affd640 1 --2- 172.21.15.6:0/2315190267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0ced00 0x7f8f6c162380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0b9830 tx=0x7f8f6002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c97cabdb0d297e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f6a7fc640 1 -- 172.21.15.6:0/2315190267 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60035020 con 0x7f8f6c0ced00 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f6a7fc640 1 -- 172.21.15.6:0/2315190267 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f60031640 con 0x7f8f6c0ced00 2024-03-18T14:41:41.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f6a7fc640 1 -- 172.21.15.6:0/2315190267 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6002d040 con 0x7f8f6c0ced00 2024-03-18T14:41:41.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f71e97640 1 -- 172.21.15.6:0/2315190267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0ced00 msgr2=0x7f8f6c162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2315190267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0ced00 0x7f8f6c162380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0b9830 tx=0x7f8f6002b8f0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f71e97640 1 -- 172.21.15.6:0/2315190267 shutdown_connections 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2315190267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c0d3210 0x7f8f6c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2315190267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0ced00 0x7f8f6c162380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f71e97640 1 -- 172.21.15.6:0/2315190267 >> 172.21.15.6:0/2315190267 conn(0x7f8f6c072ce0 msgr2=0x7f8f6c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.046+0000 7f8f71e97640 1 -- 172.21.15.6:0/2315190267 shutdown_connections 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.047+0000 7f8f71e97640 1 -- 172.21.15.6:0/2315190267 wait complete. 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 Processor -- start 2024-03-18T14:41:41.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 -- start start 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f6c0ced00 0x7f8f6c0c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0d3210 0x7f8f6c1499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c15e020 0x7f8f6c0cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c0cf2c0 con 0x7f8f6c0d3210 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c0ce890 con 0x7f8f6c0ced00 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.048+0000 7f8f71e97640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6c0cf440 con 0x7f8f6c15e020 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0d3210 0x7f8f6c1499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0d3210 0x7f8f6c1499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42954/0 (socket says 172.21.15.6:42954) 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 -- 172.21.15.6:0/2278191299 learned_addr learned my addr 172.21.15.6:0/2278191299 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6b7fe640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f6c0ced00 0x7f8f6c0c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6bfff640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c15e020 0x7f8f6c0cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 -- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c15e020 msgr2=0x7f8f6c0cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c15e020 0x7f8f6c0cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 -- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f6c0ced00 msgr2=0x7f8f6c0c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f6c0ced00 0x7f8f6c0c9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 -- 172.21.15.6:0/2278191299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c149f30 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f6affd640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0d3210 0x7f8f6c1499f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f60000c00 tx=0x7f8f600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f53fff640 1 -- 172.21.15.6:0/2278191299 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60040070 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f53fff640 1 -- 172.21.15.6:0/2278191299 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f600399f0 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6c14a1c0 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f53fff640 1 -- 172.21.15.6:0/2278191299 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60035040 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.049+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f6c14a780 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.050+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f6c08afe0 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.051+0000 7f8f53fff640 1 -- 172.21.15.6:0/2278191299 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8f60038020 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.051+0000 7f8f53fff640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8f5806bbd0 0x7f8f5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.051+0000 7f8f53fff640 1 -- 172.21.15.6:0/2278191299 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8f600ac640 con 0x7f8f6c0d3210 2024-03-18T14:41:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.051+0000 7f8f6b7fe640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8f5806bbd0 0x7f8f5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.052+0000 7f8f6b7fe640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8f5806bbd0 0x7f8f5806e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c1f76b0 tx=0x7f8f5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.056+0000 7f8f53fff640 1 -- 172.21.15.6:0/2278191299 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8f6007a7a0 con 0x7f8f6c0d3210 2024-03-18T14:41:41.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.239+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f6c04f620 con 0x7f8f6c0d3210 2024-03-18T14:41:41.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.240+0000 7f8f53fff640 1 -- 172.21.15.6:0/2278191299 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f8f6007e840 con 0x7f8f6c0d3210 2024-03-18T14:41:41.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:41.242 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8f5806bbd0 msgr2=0x7f8f5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8f5806bbd0 0x7f8f5806e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c1f76b0 tx=0x7f8f5c009040 comp rx=0 tx=0).stop 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0d3210 msgr2=0x7f8f6c1499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0d3210 0x7f8f6c1499f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f60000c00 tx=0x7f8f600046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 shutdown_connections 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8f5806bbd0 0x7f8f5806e090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f6c15e020 0x7f8f6c0cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f6c0d3210 0x7f8f6c1499f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 --2- 172.21.15.6:0/2278191299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f6c0ced00 0x7f8f6c0c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 >> 172.21.15.6:0/2278191299 conn(0x7f8f6c072ce0 msgr2=0x7f8f6c0b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:41.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 shutdown_connections 2024-03-18T14:41:41.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.242+0000 7f8f71e97640 1 -- 172.21.15.6:0/2278191299 wait complete. 2024-03-18T14:41:41.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:41.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.354+0000 7f881b220640 1 Processor -- start 2024-03-18T14:41:41.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.354+0000 7f881b220640 1 -- start start 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f881b220640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f8814162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f881b220640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f88140ced00 0x7f88140d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f881b220640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88140b9660 con 0x7f88140d3210 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f881b220640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f88140ce390 con 0x7f88140ced00 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f881b220640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f88140b9900 con 0x7f881415e020 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f8818fbc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f8814162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f8813fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f88140ced00 0x7f88140d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f88197bd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f88140d3210 0x7f88140d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:50654/0 (socket says 172.21.15.6:50654) 2024-03-18T14:41:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f8818fbc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f8814162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46796/0 (socket says 172.21.15.6:46796) 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f88197bd640 1 -- 172.21.15.6:0/3936921442 learned_addr learned my addr 172.21.15.6:0/3936921442 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f88137fe640 1 -- 172.21.15.6:0/3936921442 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963037659 0 0) 0x7f88140b9660 con 0x7f88140d3210 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f8818fbc640 1 -- 172.21.15.6:0/3936921442 >> v2:172.21.15.92:3300/0 conn(0x7f88140ced00 msgr2=0x7f88140d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f8818fbc640 1 --2- 172.21.15.6:0/3936921442 >> v2:172.21.15.92:3300/0 conn(0x7f88140ced00 0x7f88140d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f8818fbc640 1 -- 172.21.15.6:0/3936921442 >> v1:172.21.15.92:6789/0 conn(0x7f88140d3210 legacy=0x7f88140d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.355+0000 7f8818fbc640 1 -- 172.21.15.6:0/3936921442 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88140b9be0 con 0x7f881415e020 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.356+0000 7f8818fbc640 1 --2- 172.21.15.6:0/3936921442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f8814162380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88080097a0 tx=0x7f880802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1dbccdae3611287 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.356+0000 7f88137fe640 1 -- 172.21.15.6:0/3936921442 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88080315a0 con 0x7f881415e020 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.356+0000 7f88137fe640 1 -- 172.21.15.6:0/3936921442 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8808037070 con 0x7f881415e020 2024-03-18T14:41:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.356+0000 7f88137fe640 1 -- 172.21.15.6:0/3936921442 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8808031960 con 0x7f881415e020 2024-03-18T14:41:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 -- 172.21.15.6:0/3936921442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 msgr2=0x7f8814162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 --2- 172.21.15.6:0/3936921442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f8814162380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88080097a0 tx=0x7f880802b840 comp rx=0 tx=0).stop 2024-03-18T14:41:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 -- 172.21.15.6:0/3936921442 shutdown_connections 2024-03-18T14:41:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 --2- 172.21.15.6:0/3936921442 >> v2:172.21.15.92:3300/0 conn(0x7f88140ced00 0x7f88140d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 --2- 172.21.15.6:0/3936921442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f8814162380 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 -- 172.21.15.6:0/3936921442 >> 172.21.15.6:0/3936921442 conn(0x7f8814072ce0 msgr2=0x7f88140730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 -- 172.21.15.6:0/3936921442 shutdown_connections 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.357+0000 7f881b220640 1 -- 172.21.15.6:0/3936921442 wait complete. 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.358+0000 7f881b220640 1 Processor -- start 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.358+0000 7f881b220640 1 -- start start 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f881b220640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88140ced00 0x7f88140b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f881b220640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88140d3210 0x7f88141522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f881b220640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f88141f7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f881b220640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f88140cf440 con 0x7f88140d3210 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f881b220640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88140cf2c0 con 0x7f88140ced00 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f881b220640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f88140d20a0 con 0x7f881415e020 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88140d3210 0x7f88141522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8818fbc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88140ced00 0x7f88140b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88140d3210 0x7f88141522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42962/0 (socket says 172.21.15.6:42962) 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 -- 172.21.15.6:0/750991273 learned_addr learned my addr 172.21.15.6:0/750991273 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f88197bd640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f88141f7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 -- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 msgr2=0x7f88141f7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f88141f7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 -- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88140ced00 msgr2=0x7f88140b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88140ced00 0x7f88140b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 -- 172.21.15.6:0/750991273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88141f8060 con 0x7f88140d3210 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f8813fff640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88140d3210 0x7f88141522b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f880000e8a0 tx=0x7f880000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.359+0000 7f88117fa640 1 -- 172.21.15.6:0/750991273 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8800017070 con 0x7f88140d3210 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.360+0000 7f88117fa640 1 -- 172.21.15.6:0/750991273 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880000c8e0 con 0x7f88140d3210 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.360+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88141f82e0 con 0x7f88140d3210 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.360+0000 7f88117fa640 1 -- 172.21.15.6:0/750991273 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880000ccd0 con 0x7f88140d3210 2024-03-18T14:41:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.360+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88141f88a0 con 0x7f88140d3210 2024-03-18T14:41:41.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.360+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f881408afe0 con 0x7f88140d3210 2024-03-18T14:41:41.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.361+0000 7f88117fa640 1 -- 172.21.15.6:0/750991273 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f880001a070 con 0x7f88140d3210 2024-03-18T14:41:41.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.361+0000 7f88117fa640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f87f806bbd0 0x7f87f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.361+0000 7f88117fa640 1 -- 172.21.15.6:0/750991273 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f880008aab0 con 0x7f88140d3210 2024-03-18T14:41:41.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.361+0000 7f8818fbc640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f87f806bbd0 0x7f87f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.362+0000 7f8818fbc640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f87f806bbd0 0x7f87f806e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8808036000 tx=0x7f8808008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.366+0000 7f88117fa640 1 -- 172.21.15.6:0/750991273 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f88141f8060 con 0x7f88140d3210 2024-03-18T14:41:41.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.548+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f881404f620 con 0x7f88140d3210 2024-03-18T14:41:41.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.549+0000 7f88117fa640 1 -- 172.21.15.6:0/750991273 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f8800058ca0 con 0x7f88140d3210 2024-03-18T14:41:41.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:41.551 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f87f806bbd0 msgr2=0x7f87f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f87f806bbd0 0x7f87f806e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8808036000 tx=0x7f8808008030 comp rx=0 tx=0).stop 2024-03-18T14:41:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88140d3210 msgr2=0x7f88141522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88140d3210 0x7f88141522b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f880000e8a0 tx=0x7f880000ed60 comp rx=0 tx=0).stop 2024-03-18T14:41:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 shutdown_connections 2024-03-18T14:41:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f87f806bbd0 0x7f87f806e090 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f881415e020 0x7f88141f7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88140d3210 0x7f88141522b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 --2- 172.21.15.6:0/750991273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88140ced00 0x7f88140b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 >> 172.21.15.6:0/750991273 conn(0x7f8814072ce0 msgr2=0x7f881415c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 shutdown_connections 2024-03-18T14:41:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.551+0000 7f881b220640 1 -- 172.21.15.6:0/750991273 wait complete. 2024-03-18T14:41:41.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:41:41.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.678+0000 7f5981eb2640 1 Processor -- start 2024-03-18T14:41:41.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f5981eb2640 1 -- start start 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f5981eb2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f597c15bd40 0x7f597c0d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f5981eb2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f597c0cfea0 0x7f597c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f5981eb2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597c0cc870 con 0x7f597c0d1d90 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f5981eb2640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f597c0d02d0 con 0x7f597c15bd40 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f5981eb2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f597c0d0500 con 0x7f597c0cfea0 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f597b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f597c15bd40 0x7f597c0d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f597affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f597c0d1d90 0x7f597c0cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42934/0 (socket says 172.21.15.6:42934) 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f597b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f597c15bd40 0x7f597c0d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42978/0 (socket says 172.21.15.6:42978) 2024-03-18T14:41:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.679+0000 7f597affd640 1 -- 172.21.15.6:0/2783378717 learned_addr learned my addr 172.21.15.6:0/2783378717 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.680+0000 7f597bfff640 1 --2- 172.21.15.6:0/2783378717 >> v2:172.21.15.92:3300/0 conn(0x7f597c0cfea0 0x7f597c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.680+0000 7f597b7fe640 1 -- 172.21.15.6:0/2783378717 >> v2:172.21.15.92:3300/0 conn(0x7f597c0cfea0 msgr2=0x7f597c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.680+0000 7f597b7fe640 1 --2- 172.21.15.6:0/2783378717 >> v2:172.21.15.92:3300/0 conn(0x7f597c0cfea0 0x7f597c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.680+0000 7f597b7fe640 1 -- 172.21.15.6:0/2783378717 >> v1:172.21.15.6:6789/0 conn(0x7f597c0d1d90 legacy=0x7f597c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.680+0000 7f597a7fc640 1 -- 172.21.15.6:0/2783378717 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778537455 0 0) 0x7f597c0cc870 con 0x7f597c0d1d90 2024-03-18T14:41:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.680+0000 7f597b7fe640 1 -- 172.21.15.6:0/2783378717 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c0d07e0 con 0x7f597c15bd40 2024-03-18T14:41:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.680+0000 7f597b7fe640 1 --2- 172.21.15.6:0/2783378717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0d1850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f597c0cca40 tx=0x7f596c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d9642f651ebf76 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f597a7fc640 1 -- 172.21.15.6:0/2783378717 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c035020 con 0x7f597c15bd40 2024-03-18T14:41:41.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f597a7fc640 1 -- 172.21.15.6:0/2783378717 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f596c031830 con 0x7f597c15bd40 2024-03-18T14:41:41.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f597a7fc640 1 -- 172.21.15.6:0/2783378717 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c031c00 con 0x7f597c15bd40 2024-03-18T14:41:41.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 -- 172.21.15.6:0/2783378717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 msgr2=0x7f597c0d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 --2- 172.21.15.6:0/2783378717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0d1850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f597c0cca40 tx=0x7f596c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 -- 172.21.15.6:0/2783378717 shutdown_connections 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 --2- 172.21.15.6:0/2783378717 >> v2:172.21.15.92:3300/0 conn(0x7f597c0cfea0 0x7f597c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 --2- 172.21.15.6:0/2783378717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0d1850 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 -- 172.21.15.6:0/2783378717 >> 172.21.15.6:0/2783378717 conn(0x7f597c09ae20 msgr2=0x7f597c08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 -- 172.21.15.6:0/2783378717 shutdown_connections 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.681+0000 7f5981eb2640 1 -- 172.21.15.6:0/2783378717 wait complete. 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.682+0000 7f5981eb2640 1 Processor -- start 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f5981eb2640 1 -- start start 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f5981eb2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f597c0cfea0 0x7f597c0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f5981eb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f597c0d1d90 0x7f597c14bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f5981eb2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f5981eb2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f597c0d0ac0 con 0x7f597c15bd40 2024-03-18T14:41:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f5981eb2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f597c0d2210 con 0x7f597c0cfea0 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f5981eb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f597c0d0c40 con 0x7f597c0d1d90 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f597bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f597bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42994/0 (socket says 172.21.15.6:42994) 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f597bfff640 1 -- 172.21.15.6:0/4257960241 learned_addr learned my addr 172.21.15.6:0/4257960241 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f597b7fe640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f597c0cfea0 0x7f597c0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.683+0000 7f597affd640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f597c0d1d90 0x7f597c14bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f597bfff640 1 -- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f597c0d1d90 msgr2=0x7f597c14bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f597bfff640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f597c0d1d90 0x7f597c14bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f597bfff640 1 -- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f597c0cfea0 msgr2=0x7f597c0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f597bfff640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f597c0cfea0 0x7f597c0ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f597bfff640 1 -- 172.21.15.6:0/4257960241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c0b08b0 con 0x7f597c15bd40 2024-03-18T14:41:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f597bfff640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0aff60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f597400efc0 tx=0x7f597400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f5963fff640 1 -- 172.21.15.6:0/4257960241 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5974018020 con 0x7f597c15bd40 2024-03-18T14:41:41.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f5963fff640 1 -- 172.21.15.6:0/4257960241 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5974014480 con 0x7f597c15bd40 2024-03-18T14:41:41.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597c0b0b90 con 0x7f597c15bd40 2024-03-18T14:41:41.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f5963fff640 1 -- 172.21.15.6:0/4257960241 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5974014870 con 0x7f597c15bd40 2024-03-18T14:41:41.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.684+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597c0b1150 con 0x7f597c15bd40 2024-03-18T14:41:41.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.685+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f597c14c4c0 con 0x7f597c15bd40 2024-03-18T14:41:41.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.685+0000 7f5963fff640 1 -- 172.21.15.6:0/4257960241 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5974005d10 con 0x7f597c15bd40 2024-03-18T14:41:41.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.686+0000 7f5963fff640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f596806bbd0 0x7f596806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:41.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.686+0000 7f5963fff640 1 -- 172.21.15.6:0/4257960241 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f597408ba60 con 0x7f597c15bd40 2024-03-18T14:41:41.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.686+0000 7f597b7fe640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f596806bbd0 0x7f596806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:41.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.687+0000 7f597b7fe640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f596806bbd0 0x7f596806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f596c036000 tx=0x7f596c03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:41.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.690+0000 7f5963fff640 1 -- 172.21.15.6:0/4257960241 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5974022030 con 0x7f597c15bd40 2024-03-18T14:41:41.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.910+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f597c09fa90 con 0x7f597c15bd40 2024-03-18T14:41:41.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.911+0000 7f5963fff640 1 -- 172.21.15.6:0/4257960241 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5168 (secure 0 0 0) 0x7f5974010040 con 0x7f597c15bd40 2024-03-18T14:41:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:41.913 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:39.929725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:41:41.913 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 6 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f596806bbd0 msgr2=0x7f596806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f596806bbd0 0x7f596806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f596c036000 tx=0x7f596c03c040 comp rx=0 tx=0).stop 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 msgr2=0x7f597c0aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0aff60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f597400efc0 tx=0x7f597400c490 comp rx=0 tx=0).stop 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 shutdown_connections 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f596806bbd0 0x7f596806e090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f597c15bd40 0x7f597c0aff60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f597c0d1d90 0x7f597c14bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 --2- 172.21.15.6:0/4257960241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f597c0cfea0 0x7f597c0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 >> 172.21.15.6:0/4257960241 conn(0x7f597c09ae20 msgr2=0x7f597c0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 shutdown_connections 2024-03-18T14:41:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:41.913+0000 7f5981eb2640 1 -- 172.21.15.6:0/4257960241 wait complete. 2024-03-18T14:41:41.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.024+0000 7f3300f77640 1 Processor -- start 2024-03-18T14:41:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.024+0000 7f3300f77640 1 -- start start 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f3300f77640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32fc06d3f0 0x7f32fc0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f3300f77640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc0b1b50 0x7f32fc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f3300f77640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32fc0b28b0 con 0x7f32fc15def0 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f3300f77640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f32fc0d0540 con 0x7f32fc06d3f0 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f3300f77640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f32fc0b2b90 con 0x7f32fc0b1b50 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f32f9d9b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32fc06d3f0 0x7f32fc0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f32fa59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f32fc15def0 0x7f32fc06cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42944/0 (socket says 172.21.15.6:42944) 2024-03-18T14:41:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f32fa59c640 1 -- 172.21.15.6:0/1953688740 learned_addr learned my addr 172.21.15.6:0/1953688740 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.025+0000 7f32fad9d640 1 --2- 172.21.15.6:0/1953688740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc0b1b50 0x7f32fc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f9d9b640 1 -- 172.21.15.6:0/1953688740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc0b1b50 msgr2=0x7f32fc0b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f9d9b640 1 --2- 172.21.15.6:0/1953688740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc0b1b50 0x7f32fc0b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f9d9b640 1 -- 172.21.15.6:0/1953688740 >> v1:172.21.15.6:6789/0 conn(0x7f32fc15def0 legacy=0x7f32fc06cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f9d9b640 1 -- 172.21.15.6:0/1953688740 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc0d01d0 con 0x7f32fc06d3f0 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f959a640 1 -- 172.21.15.6:0/1953688740 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150608031 0 0) 0x7f32fc0b28b0 con 0x7f32fc15def0 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f9d9b640 1 --2- 172.21.15.6:0/1953688740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc0cc1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f32e40097a0 tx=0x7f32e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a15a48b9d9391c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f959a640 1 -- 172.21.15.6:0/1953688740 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4035020 con 0x7f32fc06d3f0 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f959a640 1 -- 172.21.15.6:0/1953688740 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e4031830 con 0x7f32fc06d3f0 2024-03-18T14:41:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.026+0000 7f32f959a640 1 -- 172.21.15.6:0/1953688740 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4031c00 con 0x7f32fc06d3f0 2024-03-18T14:41:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 -- 172.21.15.6:0/1953688740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 msgr2=0x7f32fc0cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 --2- 172.21.15.6:0/1953688740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc0cc1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f32e40097a0 tx=0x7f32e402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 -- 172.21.15.6:0/1953688740 shutdown_connections 2024-03-18T14:41:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 --2- 172.21.15.6:0/1953688740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc0b1b50 0x7f32fc0b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 --2- 172.21.15.6:0/1953688740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc0cc1a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 -- 172.21.15.6:0/1953688740 >> 172.21.15.6:0/1953688740 conn(0x7f32fc09ae20 msgr2=0x7f32fc08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 -- 172.21.15.6:0/1953688740 shutdown_connections 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.027+0000 7f3300f77640 1 -- 172.21.15.6:0/1953688740 wait complete. 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.028+0000 7f3300f77640 1 Processor -- start 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f3300f77640 1 -- start start 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f3300f77640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc15f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f3300f77640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32fc0b1b50 0x7f32fc153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f3300f77640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc15def0 0x7f32fc0b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f3300f77640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc0b2540 con 0x7f32fc06d3f0 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f3300f77640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc15e2d0 con 0x7f32fc0b1b50 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f3300f77640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f32fc0d09d0 con 0x7f32fc15def0 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f32fa59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc15f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f32fa59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc15f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43014/0 (socket says 172.21.15.6:43014) 2024-03-18T14:41:42.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f32fa59c640 1 -- 172.21.15.6:0/597073458 learned_addr learned my addr 172.21.15.6:0/597073458 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f32fad9d640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc15def0 0x7f32fc0b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.029+0000 7f32f9d9b640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32fc0b1b50 0x7f32fc153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32fa59c640 1 -- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc15def0 msgr2=0x7f32fc0b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32fa59c640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc15def0 0x7f32fc0b61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32fa59c640 1 -- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32fc0b1b50 msgr2=0x7f32fc153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32fa59c640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32fc0b1b50 0x7f32fc153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32fa59c640 1 -- 172.21.15.6:0/597073458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc0b6710 con 0x7f32fc06d3f0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32fa59c640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc15f700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00c7f0 tx=0x7f32ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32f2ffd640 1 -- 172.21.15.6:0/597073458 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec018070 con 0x7f32fc06d3f0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32f2ffd640 1 -- 172.21.15.6:0/597073458 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32ec00a640 con 0x7f32fc06d3f0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32fc0b6990 con 0x7f32fc06d3f0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f32f2ffd640 1 -- 172.21.15.6:0/597073458 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec00aa30 con 0x7f32fc06d3f0 2024-03-18T14:41:42.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.030+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32fc0b6f50 con 0x7f32fc06d3f0 2024-03-18T14:41:42.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.031+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32fc09fa90 con 0x7f32fc06d3f0 2024-03-18T14:41:42.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.031+0000 7f32f2ffd640 1 -- 172.21.15.6:0/597073458 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f32ec01b070 con 0x7f32fc06d3f0 2024-03-18T14:41:42.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.032+0000 7f32f2ffd640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f32dc06bbd0 0x7f32dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.032+0000 7f32f9d9b640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f32dc06bbd0 0x7f32dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.032+0000 7f32f2ffd640 1 -- 172.21.15.6:0/597073458 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f32ec08a340 con 0x7f32fc06d3f0 2024-03-18T14:41:42.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.033+0000 7f32f9d9b640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f32dc06bbd0 0x7f32dc06e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32e4036000 tx=0x7f32e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.036+0000 7f32f2ffd640 1 -- 172.21.15.6:0/597073458 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f32fc0b6710 con 0x7f32fc06d3f0 2024-03-18T14:41:42.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.221+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32fc04f620 con 0x7f32fc06d3f0 2024-03-18T14:41:42.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.222+0000 7f32f2ffd640 1 -- 172.21.15.6:0/597073458 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f32ec0584a0 con 0x7f32fc06d3f0 2024-03-18T14:41:42.224 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:42.224 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f32dc06bbd0 msgr2=0x7f32dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f32dc06bbd0 0x7f32dc06e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32e4036000 tx=0x7f32e4008030 comp rx=0 tx=0).stop 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 msgr2=0x7f32fc15f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc15f700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00c7f0 tx=0x7f32ec00ccb0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 shutdown_connections 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f32dc06bbd0 0x7f32dc06e090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32fc15def0 0x7f32fc0b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32fc0b1b50 0x7f32fc153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 --2- 172.21.15.6:0/597073458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32fc06d3f0 0x7f32fc15f700 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 >> 172.21.15.6:0/597073458 conn(0x7f32fc09ae20 msgr2=0x7f32fc0b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 shutdown_connections 2024-03-18T14:41:42.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.224+0000 7f3300f77640 1 -- 172.21.15.6:0/597073458 wait complete. 2024-03-18T14:41:42.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T14:41:34.883167+0000', 'modified': '2024-03-18T14:41:39.929725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6837/2327775013', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6835', 'nonce': 2327775013}, {'type': 'v1', 'addr': '172.21.15.92:6837', 'nonce': 2327775013}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-03-18T14:41:42.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T14:41:42.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-18T14:41:42.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.742+0000 7f428b762640 1 Processor -- start 2024-03-18T14:41:42.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.742+0000 7f428b762640 1 -- start start 2024-03-18T14:41:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f428b762640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840d2fd0 0x7f42840b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f428b762640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42840d3430 con 0x7f42840b6df0 2024-03-18T14:41:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f428b762640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42840d1f30 con 0x7f428406c7a0 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f428b762640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f42840d2210 con 0x7f42840d2fd0 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4289cff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42840b6df0 0x7f42840b7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42960/0 (socket says 172.21.15.6:42960) 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4289cff640 1 -- 172.21.15.6:0/2562720461 learned_addr learned my addr 172.21.15.6:0/2562720461 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2562720461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840d2fd0 0x7f42840b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4283fff640 1 -- 172.21.15.6:0/2562720461 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895148209 0 0) 0x7f42840d3430 con 0x7f42840b6df0 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4283fff640 1 -- 172.21.15.6:0/2562720461 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f426c003520 con 0x7f42840b6df0 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4283fff640 1 -- 172.21.15.6:0/2562720461 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481384780 0 0) 0x7f42840d1f30 con 0x7f428406c7a0 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4283fff640 1 -- 172.21.15.6:0/2562720461 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42840d3430 con 0x7f428406c7a0 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4288cfd640 1 -- 172.21.15.6:0/2562720461 >> v1:172.21.15.92:6789/0 conn(0x7f428406c7a0 legacy=0x7f428406cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4288cfd640 1 -- 172.21.15.6:0/2562720461 >> v1:172.21.15.6:6789/0 conn(0x7f42840b6df0 legacy=0x7f42840b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.743+0000 7f4288cfd640 1 -- 172.21.15.6:0/2562720461 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42840d24f0 con 0x7f42840d2fd0 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.744+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2562720461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840d2fd0 0x7f42840b95e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f42780097d0 tx=0x7f427802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48903d35c90562fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.744+0000 7f4283fff640 1 -- 172.21.15.6:0/2562720461 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278034070 con 0x7f42840d2fd0 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.744+0000 7f4283fff640 1 -- 172.21.15.6:0/2562720461 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278030880 con 0x7f42840d2fd0 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.744+0000 7f4283fff640 1 -- 172.21.15.6:0/2562720461 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278030c30 con 0x7f42840d2fd0 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.745+0000 7f428b762640 1 -- 172.21.15.6:0/2562720461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840d2fd0 msgr2=0x7f42840b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.745+0000 7f428b762640 1 --2- 172.21.15.6:0/2562720461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840d2fd0 0x7f42840b95e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f42780097d0 tx=0x7f427802b9c0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.745+0000 7f428b762640 1 -- 172.21.15.6:0/2562720461 shutdown_connections 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.745+0000 7f428b762640 1 --2- 172.21.15.6:0/2562720461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840d2fd0 0x7f42840b95e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.745+0000 7f428b762640 1 -- 172.21.15.6:0/2562720461 >> 172.21.15.6:0/2562720461 conn(0x7f4284072ce0 msgr2=0x7f42840730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.745+0000 7f428b762640 1 -- 172.21.15.6:0/2562720461 shutdown_connections 2024-03-18T14:41:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.745+0000 7f428b762640 1 -- 172.21.15.6:0/2562720461 wait complete. 2024-03-18T14:41:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.746+0000 7f428b762640 1 Processor -- start 2024-03-18T14:41:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.746+0000 7f428b762640 1 -- start start 2024-03-18T14:41:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f428b762640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f428406c7a0 0x7f42840b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f428b762640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840b6df0 0x7f42840aa650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f428b762640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42840d2fd0 0x7f42840aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f428b762640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42840b9b20 con 0x7f428406c7a0 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f428b762640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42840d2ab0 con 0x7f42840d2fd0 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f428b762640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f42840b9ca0 con 0x7f42840b6df0 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f428406c7a0 0x7f42840b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f428406c7a0 0x7f42840b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43024/0 (socket says 172.21.15.6:43024) 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 -- 172.21.15.6:0/2878641333 learned_addr learned my addr 172.21.15.6:0/2878641333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840b6df0 0x7f42840aa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f4289cff640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42840d2fd0 0x7f42840aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 -- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840b6df0 msgr2=0x7f42840aa650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840b6df0 0x7f42840aa650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 -- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42840d2fd0 msgr2=0x7f42840aab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42840d2fd0 0x7f42840aab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 -- 172.21.15.6:0/2878641333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42841f49c0 con 0x7f428406c7a0 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f42894fe640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f428406c7a0 0x7f42840b1bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c7a0 tx=0x7f427c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.747+0000 7f4281ffb640 1 -- 172.21.15.6:0/2878641333 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c003b70 con 0x7f428406c7a0 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.748+0000 7f4281ffb640 1 -- 172.21.15.6:0/2878641333 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427c003d30 con 0x7f428406c7a0 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.748+0000 7f4281ffb640 1 -- 172.21.15.6:0/2878641333 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c00aa30 con 0x7f428406c7a0 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.748+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42840b5ba0 con 0x7f428406c7a0 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.748+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42840b6160 con 0x7f428406c7a0 2024-03-18T14:41:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.749+0000 7f4281ffb640 1 -- 172.21.15.6:0/2878641333 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f427c00abd0 con 0x7f428406c7a0 2024-03-18T14:41:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.749+0000 7f4281ffb640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f427006bbd0 0x7f427006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.750+0000 7f4281ffb640 1 -- 172.21.15.6:0/2878641333 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f427c08abc0 con 0x7f428406c7a0 2024-03-18T14:41:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.750+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f427006bbd0 0x7f427006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.750+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f427006bbd0 0x7f427006e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4278036000 tx=0x7f427803b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 Processor -- start 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 -- start start 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c30c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c60a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c31160 0x564cb4c60f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564cb4b99c80 con 0x564cb4c5e060 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564cb4b42aa0 con 0x564cb4c31160 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x564cb4b95d20 con 0x564cb4c5dc80 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f4288cfd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c60a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f4288cfd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c60a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43030/0 (socket says 172.21.15.6:43030) 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f4288cfd640 1 -- 172.21.15.6:0/3911238162 learned_addr learned my addr 172.21.15.6:0/3911238162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f4289cff640 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c31160 0x564cb4c60f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.751+0000 7f42894fe640 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c30c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f4288cfd640 1 -- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 msgr2=0x564cb4c30c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f4288cfd640 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c30c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f4288cfd640 1 -- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c31160 msgr2=0x564cb4c60f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f4288cfd640 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c31160 0x564cb4c60f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f4288cfd640 1 -- 172.21.15.6:0/3911238162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb4b64ff0 con 0x564cb4c5e060 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f4288cfd640 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c60a10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f427802bfd0 tx=0x7f4278030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf67bc00c96c8a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f426affd640 1 -- 172.21.15.6:0/3911238162 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278034040 con 0x564cb4c5e060 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f426affd640 1 -- 172.21.15.6:0/3911238162 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278054070 con 0x564cb4c5e060 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.752+0000 7f426affd640 1 -- 172.21.15.6:0/3911238162 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278039d20 con 0x564cb4c5e060 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 -- 172.21.15.6:0/3911238162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 msgr2=0x564cb4c60a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c60a10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f427802bfd0 tx=0x7f4278030000 comp rx=0 tx=0).stop 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 -- 172.21.15.6:0/3911238162 shutdown_connections 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c31160 0x564cb4c60f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c60a10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 --2- 172.21.15.6:0/3911238162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c30c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 -- 172.21.15.6:0/3911238162 >> 172.21.15.6:0/3911238162 conn(0x564cb4c32e90 msgr2=0x564cb4c7ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 -- 172.21.15.6:0/3911238162 shutdown_connections 2024-03-18T14:41:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 -- 172.21.15.6:0/3911238162 wait complete. 2024-03-18T14:41:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 -- start start 2024-03-18T14:41:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.753+0000 7f428e42b740 1 Processor -- start 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c567b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f428e42b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c61e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4c62390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f428e42b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564cb4c35530 con 0x564cb4c5e060 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564cb4b55ef0 con 0x564cb4c5fcd0 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x564cb4c37d70 con 0x564cb4c5dc80 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f428e42b740 10 client.0 _ll_register_callbacks cb 0x564cb4b50dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f4288cfd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c61e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f4288cfd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c61e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43042/0 (socket says 172.21.15.6:43042) 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f4288cfd640 1 -- 172.21.15.6:0/50393602 learned_addr learned my addr 172.21.15.6:0/50393602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:42.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f42894fe640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.754+0000 7f4289cff640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4c62390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f4288cfd640 1 -- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 msgr2=0x564cb4c567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f4288cfd640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f4288cfd640 1 -- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 msgr2=0x564cb4c62390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f4288cfd640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4c62390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f4288cfd640 1 -- 172.21.15.6:0/50393602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb4b62a30 con 0x564cb4c5e060 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f4289cff640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4c62390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f4288cfd640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c61e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f42780098f0 tx=0x7f4278004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1649ec7c472a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f425ffff640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f425ffff640 1 -- 172.21.15.6:0/50393602 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278034040 con 0x564cb4c5e060 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f425ffff640 1 -- 172.21.15.6:0/50393602 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4278039d30 con 0x564cb4c5e060 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4278054070 con 0x564cb4c5e060 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f425ffff640 1 -- 172.21.15.6:0/50393602 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f4278003610 con 0x564cb4c5e060 2024-03-18T14:41:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 10 client.4443 fetch_fsmap learned FSMap version 6 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x564cb48ae2d0 con 0x564cb4c5e060 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f425ffff640 1 -- 172.21.15.6:0/50393602 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278054070 con 0x564cb4c5e060 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f425ffff640 1 -- 172.21.15.6:0/50393602 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f42780037b0 con 0x564cb4c5e060 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 10 client.4443 fetch_fsmap finished waiting for FSMap version 6 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 10 client.4443 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 20 client.4443 populate_metadata read hostname 'smithi006' 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c95180 0x564cb4c99460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 4 client.4443 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.755+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 --> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cb4ba71c0 con 0x564cb4c95180 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.756+0000 7f42894fe640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c95180 0x564cb4c99460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.756+0000 7f425ffff640 10 client.4443 ms_handle_connect on v2:172.21.15.92:6835/2327775013 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.756+0000 7f42894fe640 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c95180 0x564cb4c99460 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef96d1166e9abbe6 server_cookie=b4dceb5dd44fe9e2 in_seq=0 out_seq=0 2024-03-18T14:41:42.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f425ffff640 1 -- 172.21.15.6:0/50393602 <== mds.0 v2:172.21.15.92:6835/2327775013 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cb4ba71c0 con 0x564cb4c95180 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f425ffff640 10 client.4443 handle_command_reply: tid=0 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 client.4443 shutdown 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 msgr2=0x564cb4c61e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c61e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f42780098f0 tx=0x7f4278004730 comp rx=0 tx=0).stop 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 shutdown_connections 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c95180 0x564cb4c99460 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4c62390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5e060 0x564cb4c61e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 --2- 172.21.15.6:0/50393602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5dc80 0x564cb4c567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f425ffff640 0 client.4443 ms_handle_reset on v2:172.21.15.92:6835/2327775013 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 >> 172.21.15.6:0/50393602 conn(0x564cb4c31160 msgr2=0x564cb4c34c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 shutdown_connections 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 1 -- 172.21.15.6:0/50393602 wait complete. 2024-03-18T14:41:42.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.757+0000 7f428e42b740 20 client.4443 trim_cache size 0 max 16384 2024-03-18T14:41:42.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.775+0000 7f428e42b740 1 Processor -- start 2024-03-18T14:41:42.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.775+0000 7f428e42b740 1 -- start start 2024-03-18T14:41:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5dc80 0x564cb4c61c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4c621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f428e42b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5fcd0 0x564cb4c8a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f428e42b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42842007b0 con 0x564cb4c5fcd0 2024-03-18T14:41:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42841ffb10 con 0x564cb4c5dc80 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4284200160 con 0x564cb4c5e060 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5fcd0 0x564cb4c8a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5fcd0 0x564cb4c8a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43046/0 (socket says 172.21.15.6:43046) 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 -- 172.21.15.6:0/638454964 learned_addr learned my addr 172.21.15.6:0/638454964 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4288cfd640 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4c621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f42894fe640 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5dc80 0x564cb4c61c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 -- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 msgr2=0x564cb4c621a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4c621a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 -- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5dc80 msgr2=0x564cb4c61c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5dc80 0x564cb4c61c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 -- 172.21.15.6:0/638454964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb49786b0 con 0x564cb4c5fcd0 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f42894fe640 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5dc80 0x564cb4c61c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f4289cff640 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5fcd0 0x564cb4c8a010 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42740073a0 tx=0x7f4274007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ca0e6a9dcb0d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f426a7fc640 1 -- 172.21.15.6:0/638454964 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274001ce0 con 0x564cb4c5fcd0 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f426a7fc640 1 -- 172.21.15.6:0/638454964 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4274016420 con 0x564cb4c5fcd0 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.776+0000 7f426a7fc640 1 -- 172.21.15.6:0/638454964 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274016810 con 0x564cb4c5fcd0 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 -- 172.21.15.6:0/638454964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5fcd0 msgr2=0x564cb4c8a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5fcd0 0x564cb4c8a010 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42740073a0 tx=0x7f4274007860 comp rx=0 tx=0).stop 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 -- 172.21.15.6:0/638454964 shutdown_connections 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5fcd0 0x564cb4c8a010 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4c621a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 --2- 172.21.15.6:0/638454964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5dc80 0x564cb4c61c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 -- 172.21.15.6:0/638454964 >> 172.21.15.6:0/638454964 conn(0x564cb4c31160 msgr2=0x564cb4c7ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 -- 172.21.15.6:0/638454964 shutdown_connections 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 -- 172.21.15.6:0/638454964 wait complete. 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.777+0000 7f428e42b740 1 -- start start 2024-03-18T14:41:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.778+0000 7f428e42b740 1 Processor -- start 2024-03-18T14:41:42.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.778+0000 7f428e42b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5dc80 0x564cb4c62060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.778+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4cd74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.778+0000 7f428e42b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4cd79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.778+0000 7f428e42b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564cb4c95000 con 0x564cb4c5dc80 2024-03-18T14:41:42.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.778+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564cb4c59520 con 0x564cb4c5fcd0 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.778+0000 7f428e42b740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x564cb4c955c0 con 0x564cb4c5e060 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f428e42b740 10 client.0 _ll_register_callbacks cb 0x7f4274006940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5dc80 0x564cb4c62060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5dc80 0x564cb4c62060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43060/0 (socket says 172.21.15.6:43060) 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 -- 172.21.15.6:0/2200382664 learned_addr learned my addr 172.21.15.6:0/2200382664 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4cd74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f4289cff640 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4cd79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 -- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 msgr2=0x564cb4cd74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4cd74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 -- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 msgr2=0x564cb4cd79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4cd79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 -- 172.21.15.6:0/2200382664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb4c625a0 con 0x564cb4c5dc80 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f42894fe640 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5dc80 0x564cb4c62060 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f427c058a10 tx=0x7f427c001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad316396563f8bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f425ffff640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:41:42.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f425ffff640 1 -- 172.21.15.6:0/2200382664 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c076060 con 0x564cb4c5dc80 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f425ffff640 1 -- 172.21.15.6:0/2200382664 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f427c055380 con 0x564cb4c5dc80 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4274016420 con 0x564cb4c5dc80 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f425ffff640 1 -- 172.21.15.6:0/2200382664 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f427c08a040 con 0x564cb4c5dc80 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f428e42b740 10 client.4449 fetch_fsmap learned FSMap version 6 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x564cb4c5c030 con 0x564cb4c5dc80 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.779+0000 7f425ffff640 1 -- 172.21.15.6:0/2200382664 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4274016420 con 0x564cb4c5dc80 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f425ffff640 1 -- 172.21.15.6:0/2200382664 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f427c058160 con 0x564cb4c5dc80 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 10 client.4449 fetch_fsmap finished waiting for FSMap version 6 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 10 client.4449 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 20 client.4449 populate_metadata read hostname 'smithi006' 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c305e0 0x564cb4cf8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 4 client.4449 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 --> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] -- command(tid 0: {"prefix": "status"}) -- 0x564cb4c59a10 con 0x564cb4c305e0 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c305e0 0x564cb4cf8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f425ffff640 10 client.4449 ms_handle_connect on v2:172.21.15.92:6835/2327775013 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f4288cfd640 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c305e0 0x564cb4cf8330 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19e010c6235ac913 server_cookie=492096ae5087e9e7 in_seq=0 out_seq=0 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f425ffff640 1 -- 172.21.15.6:0/2200382664 <== mds.0 v2:172.21.15.92:6835/2327775013 1 ==== command_reply(tid 0: 0 ) ==== 8+0+323 (crc 0 0 0) 0x564cb4c59a10 con 0x564cb4c305e0 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f425ffff640 10 client.4449 handle_command_reply: tid=0 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 client.4449 shutdown 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5dc80 msgr2=0x564cb4c62060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5dc80 0x564cb4c62060 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f427c058a10 tx=0x7f427c001250 comp rx=0 tx=0).stop 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 shutdown_connections 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x564cb4c305e0 0x564cb4cf8330 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564cb4c5fcd0 0x564cb4cd79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564cb4c5e060 0x564cb4cd74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 --2- 172.21.15.6:0/2200382664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564cb4c5dc80 0x564cb4c62060 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f425ffff640 0 client.4449 ms_handle_reset on v2:172.21.15.92:6835/2327775013 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.780+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 >> 172.21.15.6:0/2200382664 conn(0x564cb4c31160 msgr2=0x564cb4c8b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 shutdown_connections 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428e42b740 1 -- 172.21.15.6:0/2200382664 wait complete. 2024-03-18T14:41:42.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428e42b740 20 client.4449 trim_cache size 0 max 16384 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a75a8f99-9e80-46f7-9da4-368af92e8063", 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4294, 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 7.242654817, 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 6, 2024-03-18T14:41:42.789 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 25, 2024-03-18T14:41:42.790 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 25, 2024-03-18T14:41:42.790 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 13.827565831999999 2024-03-18T14:41:42.790 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:41:42.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f427006bbd0 msgr2=0x7f427006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f427006bbd0 0x7f427006e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4278036000 tx=0x7f427803b040 comp rx=0 tx=0).stop 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f428406c7a0 msgr2=0x7f42840b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f428406c7a0 0x7f42840b1bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c7a0 tx=0x7f427c00cc60 comp rx=0 tx=0).stop 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 shutdown_connections 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f427006bbd0 0x7f427006e090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42840d2fd0 0x7f42840aab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42840b6df0 0x7f42840aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 --2- 172.21.15.6:0/2878641333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f428406c7a0 0x7f42840b1bf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 >> 172.21.15.6:0/2878641333 conn(0x7f4284072ce0 msgr2=0x7f42840cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 shutdown_connections 2024-03-18T14:41:42.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.781+0000 7f428b762640 1 -- 172.21.15.6:0/2878641333 wait complete. 2024-03-18T14:41:42.793 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T14:41:42.793 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:42.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:41:42.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.892+0000 7f52c0d1b640 1 Processor -- start 2024-03-18T14:41:42.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.892+0000 7f52c0d1b640 1 -- start start 2024-03-18T14:41:42.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.893+0000 7f52c0d1b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f52bc15dfd0 0x7f52bc0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.893+0000 7f52c0d1b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52bc0b6560 con 0x7f52bc0d35f0 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.893+0000 7f52c0d1b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52bc0d2100 con 0x7f52bc161a30 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.893+0000 7f52c0d1b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f52bc15e3b0 con 0x7f52bc15dfd0 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.893+0000 7f52bad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f52bc0d35f0 0x7f52bc0b5e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42964/0 (socket says 172.21.15.6:42964) 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.893+0000 7f52bad9d640 1 -- 172.21.15.6:0/550603181 learned_addr learned my addr 172.21.15.6:0/550603181 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.893+0000 7f52ba59c640 1 --2- 172.21.15.6:0/550603181 >> v2:172.21.15.92:3300/0 conn(0x7f52bc15dfd0 0x7f52bc0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52b959a640 1 -- 172.21.15.6:0/550603181 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2792106652 0 0) 0x7f52bc0b6560 con 0x7f52bc0d35f0 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52b959a640 1 -- 172.21.15.6:0/550603181 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52a8003520 con 0x7f52bc0d35f0 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52b959a640 1 -- 172.21.15.6:0/550603181 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927030874 0 0) 0x7f52bc0d2100 con 0x7f52bc161a30 2024-03-18T14:41:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52b959a640 1 -- 172.21.15.6:0/550603181 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52bc0b6560 con 0x7f52bc161a30 2024-03-18T14:41:42.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52ba59c640 1 -- 172.21.15.6:0/550603181 >> v1:172.21.15.92:6789/0 conn(0x7f52bc161a30 legacy=0x7f52bc0b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52ba59c640 1 -- 172.21.15.6:0/550603181 >> v1:172.21.15.6:6789/0 conn(0x7f52bc0d35f0 legacy=0x7f52bc0b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52ba59c640 1 -- 172.21.15.6:0/550603181 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc161f60 con 0x7f52bc15dfd0 2024-03-18T14:41:42.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.894+0000 7f52ba59c640 1 --2- 172.21.15.6:0/550603181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 0x7f52bc0ce990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009d90 tx=0x7f52ac02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44cd24c6dd1f820c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52b959a640 1 -- 172.21.15.6:0/550603181 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac039070 con 0x7f52bc15dfd0 2024-03-18T14:41:42.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52b959a640 1 -- 172.21.15.6:0/550603181 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52ac02cd00 con 0x7f52bc15dfd0 2024-03-18T14:41:42.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52b959a640 1 -- 172.21.15.6:0/550603181 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac034640 con 0x7f52bc15dfd0 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52c0d1b640 1 -- 172.21.15.6:0/550603181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 msgr2=0x7f52bc0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/550603181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 0x7f52bc0ce990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009d90 tx=0x7f52ac02bf50 comp rx=0 tx=0).stop 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52c0d1b640 1 -- 172.21.15.6:0/550603181 shutdown_connections 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/550603181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 0x7f52bc0ce990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52c0d1b640 1 -- 172.21.15.6:0/550603181 >> 172.21.15.6:0/550603181 conn(0x7f52bc06c850 msgr2=0x7f52bc06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52c0d1b640 1 -- 172.21.15.6:0/550603181 shutdown_connections 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.895+0000 7f52c0d1b640 1 -- 172.21.15.6:0/550603181 wait complete. 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.896+0000 7f52c0d1b640 1 Processor -- start 2024-03-18T14:41:42.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52c0d1b640 1 -- start start 2024-03-18T14:41:42.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52c0d1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52bc0d35f0 0x7f52bc0c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52c0d1b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 0x7f52bc0cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52c0d1b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52bc161a30 0x7f52bc1e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52c0d1b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc162800 con 0x7f52bc0d35f0 2024-03-18T14:41:42.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52c0d1b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc162520 con 0x7f52bc15dfd0 2024-03-18T14:41:42.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52c0d1b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f52bc0d1030 con 0x7f52bc161a30 2024-03-18T14:41:42.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52ba59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52bc0d35f0 0x7f52bc0c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52ba59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52bc0d35f0 0x7f52bc0c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43074/0 (socket says 172.21.15.6:43074) 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52ba59c640 1 -- 172.21.15.6:0/2583613071 learned_addr learned my addr 172.21.15.6:0/2583613071 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52b9d9b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 0x7f52bc0cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.897+0000 7f52bad9d640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52bc161a30 0x7f52bc1e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52ba59c640 1 -- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52bc161a30 msgr2=0x7f52bc1e95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52ba59c640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52bc161a30 0x7f52bc1e95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52ba59c640 1 -- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 msgr2=0x7f52bc0cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52ba59c640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 0x7f52bc0cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52ba59c640 1 -- 172.21.15.6:0/2583613071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc1e83a0 con 0x7f52bc0d35f0 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52ba59c640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52bc0d35f0 0x7f52bc0c98d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f52ac000f80 tx=0x7f52ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52b2ffd640 1 -- 172.21.15.6:0/2583613071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac039050 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52b2ffd640 1 -- 172.21.15.6:0/2583613071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52ac045070 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52bc1e8680 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52b2ffd640 1 -- 172.21.15.6:0/2583613071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac009450 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.898+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52bc1e8c40 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.899+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52bc08afe0 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.899+0000 7f52b2ffd640 1 -- 172.21.15.6:0/2583613071 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f52ac03e850 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.900+0000 7f52b2ffd640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f529c06bbd0 0x7f529c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.900+0000 7f52b2ffd640 1 -- 172.21.15.6:0/2583613071 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f52ac0adaf0 con 0x7f52bc0d35f0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.900+0000 7f52b9d9b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f529c06bbd0 0x7f529c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:42.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.900+0000 7f52b9d9b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f529c06bbd0 0x7f529c06e090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f52a4007fd0 tx=0x7f52a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:42.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:42.904+0000 7f52b2ffd640 1 -- 172.21.15.6:0/2583613071 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f52ac07bd70 con 0x7f52bc0d35f0 2024-03-18T14:41:43.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.123+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f52bc0ce0f0 con 0x7f52bc0d35f0 2024-03-18T14:41:43.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.123+0000 7f52b2ffd640 1 -- 172.21.15.6:0/2583613071 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7f52ac07fe10 con 0x7f52bc0d35f0 2024-03-18T14:41:43.125 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T14:41:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f529c06bbd0 msgr2=0x7f529c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f529c06bbd0 0x7f529c06e090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f52a4007fd0 tx=0x7f52a4009040 comp rx=0 tx=0).stop 2024-03-18T14:41:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52bc0d35f0 msgr2=0x7f52bc0c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52bc0d35f0 0x7f52bc0c98d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f52ac000f80 tx=0x7f52ac004a40 comp rx=0 tx=0).stop 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 shutdown_connections 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f529c06bbd0 0x7f529c06e090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52bc161a30 0x7f52bc1e95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52bc15dfd0 0x7f52bc0cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 --2- 172.21.15.6:0/2583613071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52bc0d35f0 0x7f52bc0c98d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 >> 172.21.15.6:0/2583613071 conn(0x7f52bc06c850 msgr2=0x7f52bc0d2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 shutdown_connections 2024-03-18T14:41:43.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43.125+0000 7f52c0d1b640 1 -- 172.21.15.6:0/2583613071 wait complete. 2024-03-18T14:41:43.137 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T14:41:43.137 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:41:43.138 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:41:43.138 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T14:41:43.138 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:43.138 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T14:41:43.138 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:41:43.138 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:41:43.138 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T14:41:43.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:43.138 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:41:43.153 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:41:43.153 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:41:43.153 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:41:43.153 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:41:43.153 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:41:43.154 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42662sec preferred_lft 42662sec 2024-03-18T14:41:43.155 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:41:43.155 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:41:43.155 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:41:43.155 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:41:43.155 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:41:43.155 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T14:41:43.156 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:41:43.156 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:41:43.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-03-18T14:41:43.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-03-18T14:41:43.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-03-18T14:41:43.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T14:41:43.156 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:41:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:41:43.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:41:43.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:43.361 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:41:43.386 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:41:43.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:43.389 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:41:43.466 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:41:43.466 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:41:43.466 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:41:43.466 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:41:43.466 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:41:43.467 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:41:43.467 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:41:43.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:41:43.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:41:43.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:41:43.467 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:41:43.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:41:43.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:41:43.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:43.770 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:41:43.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:43.786 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-18T14:41:43.840 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:41:43.840 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:41:43.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T14:41:43.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T14:41:43.840 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:41:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:41:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:41:43.960 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T14:41:43.960 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:41:43.961 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:41:43.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T14:41:43.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T14:41:43.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T14:41:43.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T14:41:43.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T14:41:43.961 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:41:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:41:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:41:44.102 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:41:44.102 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:41:44.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:41:44.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T14:41:44.103 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:41:44.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:41:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:41:44.183 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T14:41:44.184 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:41:44.184 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:44.198 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T14:41:44.198 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:41:44.198 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:44.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:41:44.319 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:41:44.374 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:41:44.374 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:41:44.374 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:41:44.375 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T14:41:44.376 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T14:41:44.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:44.377 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:41:44.428 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:41:44.428 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T14:41:44.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:41:44.496 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:41:44.551 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:41:44.551 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:41:44.551 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:41:44.551 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:41:44.551 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:41:44.552 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T14:41:44.553 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T14:41:44.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:44.554 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:41:44.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:2024-03-18T14:41:44.556+0000 7f3c3124d480 -1 init, newargv = 0x563ed42b4de0 newargc=15 2024-03-18T14:41:44.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[44753]: starting ceph client 2024-03-18T14:41:44.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[44753]: starting fuse 2024-03-18T14:41:44.571 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:41:44.572 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T14:41:44.572 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:41:44.573 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:41:44.574 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:41:44.574 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.admin")) 2024-03-18T14:41:44.574 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:41:44.693 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.admin.44753.asok 2024-03-18T14:41:44.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:41:44.703 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44753.asok 2024-03-18T14:41:44.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:44.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44753.asok status 2024-03-18T14:41:44.840 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:41:44.840 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:41:44.840 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:41:44.840 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:41:44.840 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "admin", 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "44753", 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4458, 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:41:44.841 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4458 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1651412439 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1651412439 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.4458 192.168.0.1:0/1651412439", 2024-03-18T14:41:44.842 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/1651412439", 2024-03-18T14:41:44.843 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:41:44.843 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 6, 2024-03-18T14:41:44.843 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 25, 2024-03-18T14:41:44.843 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:41:44.843 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:41:44.843 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:41:44.843 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:41:44.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:44.849 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:41:44.849 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:44.863 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:41:44.864 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:44.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:44.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:44.931 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T14:41:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T14:41:45.048 INFO:teuthology.orchestra.run.smithi006.stdout:# file: . 2024-03-18T14:41:45.048 INFO:teuthology.orchestra.run.smithi006.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T14:41:45.048 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:41:45.054 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:45.054 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:41:45.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.209+0000 7fae401a9640 1 Processor -- start 2024-03-18T14:41:45.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.209+0000 7fae401a9640 1 -- start start 2024-03-18T14:41:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae401a9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fae380b1c40 0x7fae380d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae401a9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae380cfd80 con 0x7fae3806cce0 2024-03-18T14:41:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae401a9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae380b2270 con 0x7fae380b17b0 2024-03-18T14:41:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae401a9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fae380b2df0 con 0x7fae380b1c40 2024-03-18T14:41:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3df45640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fae3806cce0 0x7fae3806d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50524/0 (socket says 172.21.15.6:50524) 2024-03-18T14:41:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3df45640 1 -- 172.21.15.6:0/2324392897 learned_addr learned my addr 172.21.15.6:0/2324392897 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3e746640 1 --2- 172.21.15.6:0/2324392897 >> v2:172.21.15.92:3300/0 conn(0x7fae380b1c40 0x7fae380d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3cf43640 1 -- 172.21.15.6:0/2324392897 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396975950 0 0) 0x7fae380cfd80 con 0x7fae3806cce0 2024-03-18T14:41:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3cf43640 1 -- 172.21.15.6:0/2324392897 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae24003520 con 0x7fae3806cce0 2024-03-18T14:41:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3e746640 1 -- 172.21.15.6:0/2324392897 >> v1:172.21.15.92:6789/0 conn(0x7fae380b17b0 legacy=0x7fae380d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3e746640 1 -- 172.21.15.6:0/2324392897 >> v1:172.21.15.6:6789/0 conn(0x7fae3806cce0 legacy=0x7fae3806d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.210+0000 7fae3e746640 1 -- 172.21.15.6:0/2324392897 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae380b2550 con 0x7fae380b1c40 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.211+0000 7fae3e746640 1 --2- 172.21.15.6:0/2324392897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae380d0970 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae30009d90 tx=0x7fae3002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8335c0e6ef420842 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.211+0000 7fae3cf43640 1 -- 172.21.15.6:0/2324392897 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30039070 con 0x7fae380b1c40 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.211+0000 7fae3cf43640 1 -- 172.21.15.6:0/2324392897 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae3002cc60 con 0x7fae380b1c40 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.211+0000 7fae3cf43640 1 -- 172.21.15.6:0/2324392897 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30034640 con 0x7fae380b1c40 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.212+0000 7fae401a9640 1 -- 172.21.15.6:0/2324392897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 msgr2=0x7fae380d0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.212+0000 7fae401a9640 1 --2- 172.21.15.6:0/2324392897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae380d0970 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae30009d90 tx=0x7fae3002beb0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.212+0000 7fae401a9640 1 -- 172.21.15.6:0/2324392897 shutdown_connections 2024-03-18T14:41:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.212+0000 7fae401a9640 1 --2- 172.21.15.6:0/2324392897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae380d0970 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.212+0000 7fae401a9640 1 -- 172.21.15.6:0/2324392897 >> 172.21.15.6:0/2324392897 conn(0x7fae3808b130 msgr2=0x7fae3808b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.212+0000 7fae401a9640 1 -- 172.21.15.6:0/2324392897 shutdown_connections 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.212+0000 7fae401a9640 1 -- 172.21.15.6:0/2324392897 wait complete. 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.213+0000 7fae401a9640 1 Processor -- start 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.213+0000 7fae401a9640 1 -- start start 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae3806cce0 0x7fae3815e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae380b17b0 0x7fae3814cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae3814d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae3815e330 con 0x7fae380b17b0 2024-03-18T14:41:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fae380b2b10 con 0x7fae380b1c40 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fae3806c690 con 0x7fae3806cce0 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3d744640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae380b17b0 0x7fae3814cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3df45640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae3806cce0 0x7fae3815e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3d744640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae380b17b0 0x7fae3814cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48300/0 (socket says 172.21.15.6:48300) 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3e746640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae3814d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3d744640 1 -- 172.21.15.6:0/573500918 learned_addr learned my addr 172.21.15.6:0/573500918 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3e746640 1 -- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae3806cce0 msgr2=0x7fae3815e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3e746640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae3806cce0 0x7fae3815e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3e746640 1 -- 172.21.15.6:0/573500918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae380b17b0 msgr2=0x7fae3814cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3e746640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae380b17b0 0x7fae3814cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3e746640 1 -- 172.21.15.6:0/573500918 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae38161240 con 0x7fae380b1c40 2024-03-18T14:41:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae3e746640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae3814d0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fae30033000 tx=0x7fae30004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae2e7fc640 1 -- 172.21.15.6:0/573500918 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30042030 con 0x7fae380b1c40 2024-03-18T14:41:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae2e7fc640 1 -- 172.21.15.6:0/573500918 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae3003a040 con 0x7fae380b1c40 2024-03-18T14:41:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae380c58b0 con 0x7fae380b1c40 2024-03-18T14:41:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae2e7fc640 1 -- 172.21.15.6:0/573500918 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30039070 con 0x7fae380b1c40 2024-03-18T14:41:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.214+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae380c5e70 con 0x7fae380b1c40 2024-03-18T14:41:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.215+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae3809fa80 con 0x7fae380b1c40 2024-03-18T14:41:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.216+0000 7fae2e7fc640 1 -- 172.21.15.6:0/573500918 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fae30045070 con 0x7fae380b1c40 2024-03-18T14:41:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.216+0000 7fae2e7fc640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae1c06bbd0 0x7fae1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.216+0000 7fae2e7fc640 1 -- 172.21.15.6:0/573500918 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fae300acc10 con 0x7fae380b1c40 2024-03-18T14:41:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.216+0000 7fae3df45640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae1c06bbd0 0x7fae1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.217+0000 7fae3df45640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae1c06bbd0 0x7fae1c06e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae340011e0 tx=0x7fae34007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.221+0000 7fae2e7fc640 1 -- 172.21.15.6:0/573500918 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fae3004c070 con 0x7fae380b1c40 2024-03-18T14:41:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.404+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae380c8680 con 0x7fae380b1c40 2024-03-18T14:41:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.405+0000 7fae2e7fc640 1 -- 172.21.15.6:0/573500918 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7fae3007ada0 con 0x7fae380b1c40 2024-03-18T14:41:45.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:45.408 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae1c06bbd0 msgr2=0x7fae1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae1c06bbd0 0x7fae1c06e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae340011e0 tx=0x7fae34007d30 comp rx=0 tx=0).stop 2024-03-18T14:41:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 msgr2=0x7fae3814d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae3814d0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fae30033000 tx=0x7fae30004a40 comp rx=0 tx=0).stop 2024-03-18T14:41:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 shutdown_connections 2024-03-18T14:41:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae1c06bbd0 0x7fae1c06e090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae380b1c40 0x7fae3814d0a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae380b17b0 0x7fae3814cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 --2- 172.21.15.6:0/573500918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae3806cce0 0x7fae3815e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 >> 172.21.15.6:0/573500918 conn(0x7fae3808b130 msgr2=0x7fae3806ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:45.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 shutdown_connections 2024-03-18T14:41:45.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.407+0000 7fae401a9640 1 -- 172.21.15.6:0/573500918 wait complete. 2024-03-18T14:41:45.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:41:45.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:41:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.523+0000 7f9888e3f640 1 Processor -- start 2024-03-18T14:41:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.523+0000 7f9888e3f640 1 -- start start 2024-03-18T14:41:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f9888e3f640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f98840b1980 0x7f988415c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f9888e3f640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840d1400 0x7f98841658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f9888e3f640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9884156760 con 0x7f98840cc670 2024-03-18T14:41:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f9888e3f640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f98840b0380 con 0x7f98840b1980 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f9888e3f640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f988415c8d0 con 0x7f98840d1400 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f988259c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f98840b1980 0x7f988415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f9882d9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840d1400 0x7f98841658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f988259c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f98840b1980 0x7f988415c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:53956/0 (socket says 172.21.15.6:53956) 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f988259c640 1 -- 172.21.15.6:0/608657705 learned_addr learned my addr 172.21.15.6:0/608657705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.524+0000 7f9881d9b640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f98840cc670 0x7f98840b1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:50574/0 (socket says 172.21.15.6:50574) 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.525+0000 7f988259c640 1 -- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840d1400 msgr2=0x7f98841658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.525+0000 7f988259c640 1 --2- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840d1400 0x7f98841658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.525+0000 7f988259c640 1 -- 172.21.15.6:0/608657705 >> v1:172.21.15.92:6789/0 conn(0x7f98840cc670 legacy=0x7f98840b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.525+0000 7f988159a640 1 -- 172.21.15.6:0/608657705 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515691450 0 0) 0x7f986c001110 con 0x7f98840cc670 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.525+0000 7f988259c640 1 -- 172.21.15.6:0/608657705 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988415cbb0 con 0x7f98840b1980 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.525+0000 7f9882d9d640 1 --2- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840d1400 0x7f98841658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f988259c640 1 --2- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840b1980 0x7f988415c280 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9874009e50 tx=0x7f987402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f8189d5a4fb1e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f988159a640 1 -- 172.21.15.6:0/608657705 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987402cb00 con 0x7f98840b1980 2024-03-18T14:41:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f988159a640 1 -- 172.21.15.6:0/608657705 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f987402cca0 con 0x7f98840b1980 2024-03-18T14:41:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f988159a640 1 -- 172.21.15.6:0/608657705 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874034610 con 0x7f98840b1980 2024-03-18T14:41:45.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 -- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840b1980 msgr2=0x7f988415c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 --2- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840b1980 0x7f988415c280 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9874009e50 tx=0x7f987402bf60 comp rx=0 tx=0).stop 2024-03-18T14:41:45.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 -- 172.21.15.6:0/608657705 shutdown_connections 2024-03-18T14:41:45.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 --2- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840d1400 0x7f98841658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 --2- 172.21.15.6:0/608657705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840b1980 0x7f988415c280 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 -- 172.21.15.6:0/608657705 >> 172.21.15.6:0/608657705 conn(0x7f9884099a50 msgr2=0x7f98840d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 -- 172.21.15.6:0/608657705 shutdown_connections 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.526+0000 7f9888e3f640 1 -- 172.21.15.6:0/608657705 wait complete. 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 Processor -- start 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 -- start start 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98840b1980 0x7f98840c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840cc670 0x7f98841550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840d1400 0x7f98840cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f988415d170 con 0x7f98840b1980 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f988409a170 con 0x7f98840d1400 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f9888e3f640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f988406bb40 con 0x7f98840cc670 2024-03-18T14:41:45.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f988259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98840b1980 0x7f98840c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f988259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98840b1980 0x7f98840c38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48316/0 (socket says 172.21.15.6:48316) 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.528+0000 7f988259c640 1 -- 172.21.15.6:0/3778245266 learned_addr learned my addr 172.21.15.6:0/3778245266 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f9882d9d640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840d1400 0x7f98840cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f9881d9b640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840cc670 0x7f98841550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f988259c640 1 -- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840cc670 msgr2=0x7f98841550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f988259c640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840cc670 0x7f98841550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f988259c640 1 -- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840d1400 msgr2=0x7f98840cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f988259c640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840d1400 0x7f98840cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f988259c640 1 -- 172.21.15.6:0/3778245266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98840cf7f0 con 0x7f98840b1980 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f988259c640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98840b1980 0x7f98840c38e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9874009e50 tx=0x7f9874004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f987affd640 1 -- 172.21.15.6:0/3778245266 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874043020 con 0x7f98840b1980 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f987affd640 1 -- 172.21.15.6:0/3778245266 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f987402ce50 con 0x7f98840b1980 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98840cfad0 con 0x7f98840b1980 2024-03-18T14:41:45.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f987affd640 1 -- 172.21.15.6:0/3778245266 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874009510 con 0x7f98840b1980 2024-03-18T14:41:45.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.529+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98841f49b0 con 0x7f98840b1980 2024-03-18T14:41:45.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.530+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98840751b0 con 0x7f98840b1980 2024-03-18T14:41:45.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.531+0000 7f987affd640 1 -- 172.21.15.6:0/3778245266 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9874046020 con 0x7f98840b1980 2024-03-18T14:41:45.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.531+0000 7f987affd640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f986406bbd0 0x7f986406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.531+0000 7f987affd640 1 -- 172.21.15.6:0/3778245266 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f98740ad280 con 0x7f98840b1980 2024-03-18T14:41:45.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.531+0000 7f9881d9b640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f986406bbd0 0x7f986406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.532+0000 7f9881d9b640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f986406bbd0 0x7f986406e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f98840d0350 tx=0x7f986c007b50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.535+0000 7f987affd640 1 -- 172.21.15.6:0/3778245266 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f987404d030 con 0x7f98840b1980 2024-03-18T14:41:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.718+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f988415ea00 con 0x7f98840b1980 2024-03-18T14:41:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.718+0000 7f987affd640 1 -- 172.21.15.6:0/3778245266 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f987407b3f0 con 0x7f98840b1980 2024-03-18T14:41:45.720 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f986406bbd0 msgr2=0x7f986406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f986406bbd0 0x7f986406e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f98840d0350 tx=0x7f986c007b50 comp rx=0 tx=0).stop 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98840b1980 msgr2=0x7f98840c38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98840b1980 0x7f98840c38e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9874009e50 tx=0x7f9874004a40 comp rx=0 tx=0).stop 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 shutdown_connections 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f986406bbd0 0x7f986406e090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98840d1400 0x7f98840cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98840cc670 0x7f98841550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 --2- 172.21.15.6:0/3778245266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98840b1980 0x7f98840c38e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 >> 172.21.15.6:0/3778245266 conn(0x7f9884099a50 msgr2=0x7f988415de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:45.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 shutdown_connections 2024-03-18T14:41:45.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.721+0000 7f9888e3f640 1 -- 172.21.15.6:0/3778245266 wait complete. 2024-03-18T14:41:45.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:41:45.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.836+0000 7fd8aeec6640 1 Processor -- start 2024-03-18T14:41:45.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.836+0000 7fd8aeec6640 1 -- start start 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8aeec6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a80b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8aeec6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8a80b7e00 0x7fd8a80b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8aeec6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a815e3b0 con 0x7fd8a80b5740 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8aeec6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a80b60e0 con 0x7fd8a80b7e00 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8aeec6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a80b8470 con 0x7fd8a815dfd0 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8ad463640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8a80b7e00 0x7fd8a80b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8acc62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a80b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8ad463640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8a80b7e00 0x7fd8a80b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48328/0 (socket says 172.21.15.6:48328) 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8ad463640 1 -- 172.21.15.6:0/1693035959 learned_addr learned my addr 172.21.15.6:0/1693035959 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8ad463640 1 -- 172.21.15.6:0/1693035959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 msgr2=0x7fd8a80b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8ad463640 1 --2- 172.21.15.6:0/1693035959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a80b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8ad463640 1 -- 172.21.15.6:0/1693035959 >> v1:172.21.15.92:6789/0 conn(0x7fd8a80b5740 legacy=0x7fd8a80d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8ad463640 1 -- 172.21.15.6:0/1693035959 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a80b8750 con 0x7fd8a80b7e00 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.837+0000 7fd8a77fe640 1 -- 172.21.15.6:0/1693035959 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355090402 0 0) 0x7fd8a815e3b0 con 0x7fd8a80b5740 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.838+0000 7fd8ad463640 1 --2- 172.21.15.6:0/1693035959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a80b5ba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd890009870 tx=0x7fd89002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303e0fd3b40a826b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.838+0000 7fd8a77fe640 1 -- 172.21.15.6:0/1693035959 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890034070 con 0x7fd8a80b7e00 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.838+0000 7fd8a77fe640 1 -- 172.21.15.6:0/1693035959 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd890030840 con 0x7fd8a80b7e00 2024-03-18T14:41:45.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.838+0000 7fd8a77fe640 1 -- 172.21.15.6:0/1693035959 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890030c10 con 0x7fd8a80b7e00 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 -- 172.21.15.6:0/1693035959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 msgr2=0x7fd8a80b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/1693035959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a80b5ba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd890009870 tx=0x7fd89002b980 comp rx=0 tx=0).stop 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 -- 172.21.15.6:0/1693035959 shutdown_connections 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/1693035959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a80b5ba0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/1693035959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 -- 172.21.15.6:0/1693035959 >> 172.21.15.6:0/1693035959 conn(0x7fd8a8072d10 msgr2=0x7fd8a8073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 -- 172.21.15.6:0/1693035959 shutdown_connections 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.839+0000 7fd8aeec6640 1 -- 172.21.15.6:0/1693035959 wait complete. 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.840+0000 7fd8aeec6640 1 Processor -- start 2024-03-18T14:41:45.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.840+0000 7fd8aeec6640 1 -- start start 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8aeec6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8a80b5740 0x7fd8a81f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8aeec6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a81f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8aeec6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8aeec6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a80b9b90 con 0x7fd8a80b7e00 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8aeec6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a80b8d10 con 0x7fd8a80b5740 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8aeec6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a80d2550 con 0x7fd8a815dfd0 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a81f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a81f6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48344/0 (socket says 172.21.15.6:48344) 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 -- 172.21.15.6:0/2118539729 learned_addr learned my addr 172.21.15.6:0/2118539729 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8ad463640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8acc62640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8a80b5740 0x7fd8a81f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 -- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 msgr2=0x7fd8a815ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a815ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 -- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8a80b5740 msgr2=0x7fd8a81f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8a80b5740 0x7fd8a81f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.841+0000 7fd8a7fff640 1 -- 172.21.15.6:0/2118539729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a815fa00 con 0x7fd8a80b7e00 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.842+0000 7fd8a7fff640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a81f6710 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c890 tx=0x7fd89800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.842+0000 7fd8a57fa640 1 -- 172.21.15.6:0/2118539729 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898017070 con 0x7fd8a80b7e00 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.842+0000 7fd8a57fa640 1 -- 172.21.15.6:0/2118539729 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd898003dc0 con 0x7fd8a80b7e00 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.842+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a80c77f0 con 0x7fd8a80b7e00 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.842+0000 7fd8a57fa640 1 -- 172.21.15.6:0/2118539729 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8980052b0 con 0x7fd8a80b7e00 2024-03-18T14:41:45.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.842+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a80c8090 con 0x7fd8a80b7e00 2024-03-18T14:41:45.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.843+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a80c79c0 con 0x7fd8a80b7e00 2024-03-18T14:41:45.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.843+0000 7fd8a57fa640 1 -- 172.21.15.6:0/2118539729 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd89801a070 con 0x7fd8a80b7e00 2024-03-18T14:41:45.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.843+0000 7fd8a57fa640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd88806bbd0 0x7fd88806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:45.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.844+0000 7fd8a57fa640 1 -- 172.21.15.6:0/2118539729 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd89808a3d0 con 0x7fd8a80b7e00 2024-03-18T14:41:45.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.844+0000 7fd8acc62640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd88806bbd0 0x7fd88806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:45.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.844+0000 7fd8acc62640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd88806bbd0 0x7fd88806e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009aa0 tx=0x7fd8a0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:45.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:45.848+0000 7fd8a57fa640 1 -- 172.21.15.6:0/2118539729 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd898058680 con 0x7fd8a80b7e00 2024-03-18T14:41:46.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.029+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd8a80751b0 con 0x7fd8a80b7e00 2024-03-18T14:41:46.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.029+0000 7fd8a57fa640 1 -- 172.21.15.6:0/2118539729 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fd89805c720 con 0x7fd8a80b7e00 2024-03-18T14:41:46.031 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:41:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd88806bbd0 msgr2=0x7fd88806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd88806bbd0 0x7fd88806e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009aa0 tx=0x7fd8a0009190 comp rx=0 tx=0).stop 2024-03-18T14:41:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 msgr2=0x7fd8a81f6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a81f6710 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c890 tx=0x7fd89800cd50 comp rx=0 tx=0).stop 2024-03-18T14:41:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 shutdown_connections 2024-03-18T14:41:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd88806bbd0 0x7fd88806e090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8a815dfd0 0x7fd8a815ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8a80b7e00 0x7fd8a81f6710 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 --2- 172.21.15.6:0/2118539729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8a80b5740 0x7fd8a81f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 >> 172.21.15.6:0/2118539729 conn(0x7fd8a8072d10 msgr2=0x7fd8a80ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:46.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 shutdown_connections 2024-03-18T14:41:46.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.031+0000 7fd8aeec6640 1 -- 172.21.15.6:0/2118539729 wait complete. 2024-03-18T14:41:46.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:41:46.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.146+0000 7f3cc02a1640 1 Processor -- start 2024-03-18T14:41:46.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.146+0000 7f3cc02a1640 1 -- start start 2024-03-18T14:41:46.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.147+0000 7f3cc02a1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb815ddb0 0x7f3cb80b3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.147+0000 7f3cc02a1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cb8161a00 0x7f3cb80b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.147+0000 7f3cc02a1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb80b2fb0 con 0x7f3cb80b6230 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.147+0000 7f3cc02a1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3cb80cf3f0 con 0x7f3cb8161a00 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.147+0000 7f3cc02a1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb80b6670 con 0x7f3cb815ddb0 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.147+0000 7f3cbd83c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cb8161a00 0x7f3cb80b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbd83c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3cb8161a00 0x7f3cb80b5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48346/0 (socket says 172.21.15.6:48346) 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbd83c640 1 -- 172.21.15.6:0/1304310608 learned_addr learned my addr 172.21.15.6:0/1304310608 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbe03d640 1 --2- 172.21.15.6:0/1304310608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb815ddb0 0x7f3cb80b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbd83c640 1 -- 172.21.15.6:0/1304310608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb815ddb0 msgr2=0x7f3cb80b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbd83c640 1 --2- 172.21.15.6:0/1304310608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb815ddb0 0x7f3cb80b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbd83c640 1 -- 172.21.15.6:0/1304310608 >> v1:172.21.15.92:6789/0 conn(0x7f3cb80b6230 legacy=0x7f3cb80d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbd83c640 1 -- 172.21.15.6:0/1304310608 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb8161fc0 con 0x7f3cb8161a00 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.148+0000 7f3cbd83c640 1 --2- 172.21.15.6:0/1304310608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb8161a00 0x7f3cb80b5cf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009870 tx=0x7f3cac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c5791b2c0ed0f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cbd03b640 1 -- 172.21.15.6:0/1304310608 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac035020 con 0x7f3cb8161a00 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cbd03b640 1 -- 172.21.15.6:0/1304310608 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cac031640 con 0x7f3cb8161a00 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cbd03b640 1 -- 172.21.15.6:0/1304310608 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac02d040 con 0x7f3cb8161a00 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 -- 172.21.15.6:0/1304310608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb8161a00 msgr2=0x7f3cb80b5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/1304310608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb8161a00 0x7f3cb80b5cf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009870 tx=0x7f3cac02b8f0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 -- 172.21.15.6:0/1304310608 shutdown_connections 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/1304310608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb8161a00 0x7f3cb80b5cf0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/1304310608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb815ddb0 0x7f3cb80b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 -- 172.21.15.6:0/1304310608 >> 172.21.15.6:0/1304310608 conn(0x7f3cb8072ad0 msgr2=0x7f3cb8072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:46.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 -- 172.21.15.6:0/1304310608 shutdown_connections 2024-03-18T14:41:46.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.149+0000 7f3cc02a1640 1 -- 172.21.15.6:0/1304310608 wait complete. 2024-03-18T14:41:46.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.150+0000 7f3cc02a1640 1 Processor -- start 2024-03-18T14:41:46.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.150+0000 7f3cc02a1640 1 -- start start 2024-03-18T14:41:46.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cc02a1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb80b6230 0x7f3cb80cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cc02a1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb815ddb0 0x7f3cb80cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cc02a1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cb8161a00 0x7f3cb81f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cc02a1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb8162860 con 0x7f3cb815ddb0 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cc02a1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb8162580 con 0x7f3cb8161a00 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cc02a1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb815e190 con 0x7f3cb80b6230 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cbd83c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb815ddb0 0x7f3cb80cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cbd83c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb815ddb0 0x7f3cb80cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48362/0 (socket says 172.21.15.6:48362) 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cbd83c640 1 -- 172.21.15.6:0/810396976 learned_addr learned my addr 172.21.15.6:0/810396976 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cbe03d640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb80b6230 0x7f3cb80cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.151+0000 7f3cbe83e640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cb8161a00 0x7f3cb81f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cbd83c640 1 -- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb80b6230 msgr2=0x7f3cb80cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cbd83c640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb80b6230 0x7f3cb80cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cbd83c640 1 -- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cb8161a00 msgr2=0x7f3cb81f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cbd83c640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cb8161a00 0x7f3cb81f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cbd83c640 1 -- 172.21.15.6:0/810396976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb80cccc0 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cbe03d640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb80b6230 0x7f3cb80cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cbd83c640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb815ddb0 0x7f3cb80cba60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3cac000c00 tx=0x7f3cac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cb6ffd640 1 -- 172.21.15.6:0/810396976 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac040070 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cb6ffd640 1 -- 172.21.15.6:0/810396976 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cac0399f0 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cb6ffd640 1 -- 172.21.15.6:0/810396976 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac035040 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb80ccfa0 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.152+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb80cd560 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.153+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb80751b0 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.154+0000 7f3cb6ffd640 1 -- 172.21.15.6:0/810396976 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3cac04b060 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.154+0000 7f3cb6ffd640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ca406bbd0 0x7f3ca406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.154+0000 7f3cb6ffd640 1 -- 172.21.15.6:0/810396976 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3cac0ac490 con 0x7f3cb815ddb0 2024-03-18T14:41:46.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.154+0000 7f3cbe03d640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ca406bbd0 0x7f3ca406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.154+0000 7f3cbe03d640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ca406bbd0 0x7f3ca406e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009f30 tx=0x7f3cb0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.158+0000 7f3cb6ffd640 1 -- 172.21.15.6:0/810396976 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3cac07a5f0 con 0x7f3cb815ddb0 2024-03-18T14:41:46.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.340+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3cb806dac0 con 0x7f3cb815ddb0 2024-03-18T14:41:46.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.340+0000 7f3cb6ffd640 1 -- 172.21.15.6:0/810396976 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f3cac07e690 con 0x7f3cb815ddb0 2024-03-18T14:41:46.342 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:41:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ca406bbd0 msgr2=0x7f3ca406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ca406bbd0 0x7f3ca406e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009f30 tx=0x7f3cb0009190 comp rx=0 tx=0).stop 2024-03-18T14:41:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb815ddb0 msgr2=0x7f3cb80cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb815ddb0 0x7f3cb80cba60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3cac000c00 tx=0x7f3cac0046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 shutdown_connections 2024-03-18T14:41:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ca406bbd0 0x7f3ca406e090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cb8161a00 0x7f3cb81f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3cb815ddb0 0x7f3cb80cba60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 --2- 172.21.15.6:0/810396976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cb80b6230 0x7f3cb80cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 >> 172.21.15.6:0/810396976 conn(0x7f3cb8072ad0 msgr2=0x7f3cb80b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:46.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 shutdown_connections 2024-03-18T14:41:46.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.342+0000 7f3cc02a1640 1 -- 172.21.15.6:0/810396976 wait complete. 2024-03-18T14:41:46.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:41:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.457+0000 7f97fa6b8640 1 Processor -- start 2024-03-18T14:41:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.457+0000 7f97fa6b8640 1 -- start start 2024-03-18T14:41:46.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97fa6b8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97f415e030 0x7f97f40d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97fa6b8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f97f40d0450 0x7f97f40d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97fa6b8640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f40b65a0 con 0x7f97f40b6140 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97fa6b8640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f97f415e410 con 0x7f97f415e030 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97fa6b8640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f97f40ce300 con 0x7f97f40d0450 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97f415e030 0x7f97f40d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97f415e030 0x7f97f40d3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48368/0 (socket says 172.21.15.6:48368) 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f8c55640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f97f40d0450 0x7f97f40d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f3fff640 1 -- 172.21.15.6:0/1204827179 learned_addr learned my addr 172.21.15.6:0/1204827179 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f3fff640 1 -- 172.21.15.6:0/1204827179 >> v2:172.21.15.92:3300/0 conn(0x7f97f40d0450 msgr2=0x7f97f40d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f3fff640 1 --2- 172.21.15.6:0/1204827179 >> v2:172.21.15.92:3300/0 conn(0x7f97f40d0450 0x7f97f40d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f3fff640 1 -- 172.21.15.6:0/1204827179 >> v1:172.21.15.92:6789/0 conn(0x7f97f40b6140 legacy=0x7f97f40ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f3fff640 1 -- 172.21.15.6:0/1204827179 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f40d2fd0 con 0x7f97f415e030 2024-03-18T14:41:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f2ffd640 1 -- 172.21.15.6:0/1204827179 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127300741 0 0) 0x7f97f40b65a0 con 0x7f97f40b6140 2024-03-18T14:41:46.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.458+0000 7f97f8c55640 1 --2- 172.21.15.6:0/1204827179 >> v2:172.21.15.92:3300/0 conn(0x7f97f40d0450 0x7f97f40d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:46.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.459+0000 7f97f3fff640 1 --2- 172.21.15.6:0/1204827179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 0x7f97f40d3570 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f97f40b6c20 tx=0x7f97e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c399eeba4fab88e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.459+0000 7f97f2ffd640 1 -- 172.21.15.6:0/1204827179 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4034070 con 0x7f97f415e030 2024-03-18T14:41:46.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.459+0000 7f97f2ffd640 1 -- 172.21.15.6:0/1204827179 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e4030860 con 0x7f97f415e030 2024-03-18T14:41:46.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.459+0000 7f97f2ffd640 1 -- 172.21.15.6:0/1204827179 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4030c30 con 0x7f97f415e030 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 -- 172.21.15.6:0/1204827179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 msgr2=0x7f97f40d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/1204827179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 0x7f97f40d3570 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f97f40b6c20 tx=0x7f97e402b9a0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 -- 172.21.15.6:0/1204827179 shutdown_connections 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/1204827179 >> v2:172.21.15.92:3300/0 conn(0x7f97f40d0450 0x7f97f40d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/1204827179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 0x7f97f40d3570 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 -- 172.21.15.6:0/1204827179 >> 172.21.15.6:0/1204827179 conn(0x7f97f406c8c0 msgr2=0x7f97f406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 -- 172.21.15.6:0/1204827179 shutdown_connections 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.460+0000 7f97fa6b8640 1 -- 172.21.15.6:0/1204827179 wait complete. 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.461+0000 7f97fa6b8640 1 Processor -- start 2024-03-18T14:41:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.461+0000 7f97fa6b8640 1 -- start start 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f40b6140 0x7f97f40c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97f40d0450 0x7f97f4158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 0x7f97f41f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f97f40ce550 con 0x7f97f415e030 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97f40d3290 con 0x7f97f40b6140 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f97f40ce830 con 0x7f97f40d0450 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f8c55640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 0x7f97f41f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f40b6140 0x7f97f40c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f40b6140 0x7f97f40c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:53996/0 (socket says 172.21.15.6:53996) 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 -- 172.21.15.6:0/3099996035 learned_addr learned my addr 172.21.15.6:0/3099996035 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f37fe640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97f40d0450 0x7f97f4158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 -- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97f40d0450 msgr2=0x7f97f4158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97f40d0450 0x7f97f4158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 -- 172.21.15.6:0/3099996035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 msgr2=0x7f97f41f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 0x7f97f41f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 -- 172.21.15.6:0/3099996035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f40ca690 con 0x7f97f40b6140 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f3fff640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f40b6140 0x7f97f40c9760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f97e40090b0 tx=0x7f97e40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f0ff9640 1 -- 172.21.15.6:0/3099996035 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4034040 con 0x7f97f40b6140 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f0ff9640 1 -- 172.21.15.6:0/3099996035 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e403ab90 con 0x7f97f40b6140 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f40c9df0 con 0x7f97f40b6140 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97f0ff9640 1 -- 172.21.15.6:0/3099996035 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4004030 con 0x7f97f40b6140 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.462+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f40ca3b0 con 0x7f97f40b6140 2024-03-18T14:41:46.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.463+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f40751b0 con 0x7f97f40b6140 2024-03-18T14:41:46.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.464+0000 7f97f0ff9640 1 -- 172.21.15.6:0/3099996035 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f97e4037070 con 0x7f97f40b6140 2024-03-18T14:41:46.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.464+0000 7f97f0ff9640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f97e006bbd0 0x7f97e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.464+0000 7f97f0ff9640 1 -- 172.21.15.6:0/3099996035 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f97e40ace20 con 0x7f97f40b6140 2024-03-18T14:41:46.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.464+0000 7f97f37fe640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f97e006bbd0 0x7f97e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.465+0000 7f97f37fe640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f97e006bbd0 0x7f97e006e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97f40b3060 tx=0x7f97e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.469+0000 7f97f0ff9640 1 -- 172.21.15.6:0/3099996035 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f97e403f070 con 0x7f97f40b6140 2024-03-18T14:41:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.651+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f97f40d19f0 con 0x7f97f40b6140 2024-03-18T14:41:46.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.651+0000 7f97f0ff9640 1 -- 172.21.15.6:0/3099996035 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f97e407b0a0 con 0x7f97f40b6140 2024-03-18T14:41:46.653 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:41:46.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f97e006bbd0 msgr2=0x7f97e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f97e006bbd0 0x7f97e006e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97f40b3060 tx=0x7f97e8009040 comp rx=0 tx=0).stop 2024-03-18T14:41:46.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f40b6140 msgr2=0x7f97f40c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f40b6140 0x7f97f40c9760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f97e40090b0 tx=0x7f97e40046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 shutdown_connections 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f97e006bbd0 0x7f97e006e090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f415e030 0x7f97f41f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97f40d0450 0x7f97f4158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 --2- 172.21.15.6:0/3099996035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f40b6140 0x7f97f40c9760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 >> 172.21.15.6:0/3099996035 conn(0x7f97f406c8c0 msgr2=0x7f97f40b7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 shutdown_connections 2024-03-18T14:41:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.653+0000 7f97fa6b8640 1 -- 172.21.15.6:0/3099996035 wait complete. 2024-03-18T14:41:46.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:41:46.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.767+0000 7f9001eea640 1 Processor -- start 2024-03-18T14:41:46.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.767+0000 7f9001eea640 1 -- start start 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.767+0000 7f9001eea640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8ffc0d2fd0 0x7f8ffc0b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.767+0000 7f9001eea640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.767+0000 7f9001eea640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ffc0d3430 con 0x7f8ffc06c7a0 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.767+0000 7f9001eea640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f8ffc0d1f30 con 0x7f8ffc0d2fd0 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.767+0000 7f9001eea640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8ffc0d2210 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffbfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffaffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8ffc0d2fd0 0x7f8ffc0b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffb7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8ffc06c7a0 0x7f8ffc06cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:50598/0 (socket says 172.21.15.6:50598) 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffbfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50518/0 (socket says 172.21.15.6:50518) 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffb7fe640 1 -- 172.21.15.6:0/2373524784 learned_addr learned my addr 172.21.15.6:0/2373524784 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffa7fc640 1 -- 172.21.15.6:0/2373524784 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147766374 0 0) 0x7f8ffc0d3430 con 0x7f8ffc06c7a0 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffa7fc640 1 -- 172.21.15.6:0/2373524784 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fe4003520 con 0x7f8ffc06c7a0 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffaffd640 1 -- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0b6df0 msgr2=0x7f8ffc0b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffaffd640 1 --2- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffaffd640 1 -- 172.21.15.6:0/2373524784 >> v1:172.21.15.92:6789/0 conn(0x7f8ffc06c7a0 legacy=0x7f8ffc06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffaffd640 1 -- 172.21.15.6:0/2373524784 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc0d24f0 con 0x7f8ffc0d2fd0 2024-03-18T14:41:46.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.768+0000 7f8ffbfff640 1 --2- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:46.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f8ffaffd640 1 --2- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0d2fd0 0x7f8ffc0b95e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097a0 tx=0x7f8ff002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b47252ea297c7bec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f8ffa7fc640 1 -- 172.21.15.6:0/2373524784 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0034070 con 0x7f8ffc0d2fd0 2024-03-18T14:41:46.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f8ffa7fc640 1 -- 172.21.15.6:0/2373524784 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff0030610 con 0x7f8ffc0d2fd0 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f8ffa7fc640 1 -- 172.21.15.6:0/2373524784 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00309c0 con 0x7f8ffc0d2fd0 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 -- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0d2fd0 msgr2=0x7f8ffc0b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 --2- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0d2fd0 0x7f8ffc0b95e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097a0 tx=0x7f8ff002b960 comp rx=0 tx=0).stop 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 -- 172.21.15.6:0/2373524784 shutdown_connections 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 --2- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 --2- 172.21.15.6:0/2373524784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0d2fd0 0x7f8ffc0b95e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 -- 172.21.15.6:0/2373524784 >> 172.21.15.6:0/2373524784 conn(0x7f8ffc072ce0 msgr2=0x7f8ffc0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 -- 172.21.15.6:0/2373524784 shutdown_connections 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.769+0000 7f9001eea640 1 -- 172.21.15.6:0/2373524784 wait complete. 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.770+0000 7f9001eea640 1 Processor -- start 2024-03-18T14:41:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.771+0000 7f9001eea640 1 -- start start 2024-03-18T14:41:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.771+0000 7f9001eea640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06c7a0 0x7f8ffc0ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.771+0000 7f9001eea640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.771+0000 7f9001eea640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0d2fd0 0x7f8ffc1f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f9001eea640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc0b9b20 con 0x7f8ffc06c7a0 2024-03-18T14:41:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f9001eea640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc0d2ab0 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f9001eea640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8ffc0b9ca0 con 0x7f8ffc0d2fd0 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06c7a0 0x7f8ffc0ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06c7a0 0x7f8ffc0ac4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48378/0 (socket says 172.21.15.6:48378) 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffb7fe640 1 -- 172.21.15.6:0/3379585509 learned_addr learned my addr 172.21.15.6:0/3379585509 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffaffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffbfff640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0d2fd0 0x7f8ffc1f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffaffd640 1 -- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0d2fd0 msgr2=0x7f8ffc1f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffaffd640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0d2fd0 0x7f8ffc1f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffaffd640 1 -- 172.21.15.6:0/3379585509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06c7a0 msgr2=0x7f8ffc0ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffaffd640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06c7a0 0x7f8ffc0ac4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffaffd640 1 -- 172.21.15.6:0/3379585509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc0ace10 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffb7fe640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06c7a0 0x7f8ffc0ac4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8ffaffd640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b8520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0033000 tx=0x7f8ff00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8fe3fff640 1 -- 172.21.15.6:0/3379585509 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0034040 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8fe3fff640 1 -- 172.21.15.6:0/3379585509 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff003aaa0 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ffc1f6d90 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f8fe3fff640 1 -- 172.21.15.6:0/3379585509 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0006e10 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.772+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ffc1f7350 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.773+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ffc04f5b0 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.774+0000 7f8fe3fff640 1 -- 172.21.15.6:0/3379585509 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8ff004b060 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.774+0000 7f8fe3fff640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8fe806bbd0 0x7f8fe806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:46.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.774+0000 7f8fe3fff640 1 -- 172.21.15.6:0/3379585509 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8ff00ac8f0 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.774+0000 7f8ffb7fe640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8fe806bbd0 0x7f8fe806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:46.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.775+0000 7f8ffb7fe640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8fe806bbd0 0x7f8fe806e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8fec005880 tx=0x7f8fec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.779+0000 7f8fe3fff640 1 -- 172.21.15.6:0/3379585509 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8ff003f070 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.972+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8ffc0731f0 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.972+0000 7f8fe3fff640 1 -- 172.21.15.6:0/3379585509 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f8ff007b090 con 0x7f8ffc0b6df0 2024-03-18T14:41:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8fe806bbd0 msgr2=0x7f8fe806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8fe806bbd0 0x7f8fe806e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8fec005880 tx=0x7f8fec009040 comp rx=0 tx=0).stop 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0b6df0 msgr2=0x7f8ffc0b8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b8520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0033000 tx=0x7f8ff00046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 shutdown_connections 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8fe806bbd0 0x7f8fe806e090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ffc0d2fd0 0x7f8ffc1f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ffc0b6df0 0x7f8ffc0b8520 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 --2- 172.21.15.6:0/3379585509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06c7a0 0x7f8ffc0ac4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 >> 172.21.15.6:0/3379585509 conn(0x7f8ffc072ce0 msgr2=0x7f8ffc15d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 shutdown_connections 2024-03-18T14:41:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:46.975+0000 7f9001eea640 1 -- 172.21.15.6:0/3379585509 wait complete. 2024-03-18T14:41:46.987 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:41:46.987 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:41:47.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:47.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.142+0000 7f90c2977640 1 Processor -- start 2024-03-18T14:41:47.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.142+0000 7f90c2977640 1 -- start start 2024-03-18T14:41:47.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c2977640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc15e020 0x7f90bc162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c2977640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90bc0d3210 0x7f90bc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c2977640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90bc0b9660 con 0x7f90bc0ced00 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c2977640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f90bc0d2270 con 0x7f90bc0d3210 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c2977640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f90bc0ce390 con 0x7f90bc15e020 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c0f14640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90bc0d3210 0x7f90bc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90bb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f90bc0ced00 0x7f90bc0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50526/0 (socket says 172.21.15.6:50526) 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90bb7fe640 1 -- 172.21.15.6:0/791207178 learned_addr learned my addr 172.21.15.6:0/791207178 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90bbfff640 1 --2- 172.21.15.6:0/791207178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc15e020 0x7f90bc162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c0f14640 1 -- 172.21.15.6:0/791207178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc15e020 msgr2=0x7f90bc162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c0f14640 1 --2- 172.21.15.6:0/791207178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc15e020 0x7f90bc162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c0f14640 1 -- 172.21.15.6:0/791207178 >> v1:172.21.15.6:6789/0 conn(0x7f90bc0ced00 legacy=0x7f90bc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90baffd640 1 -- 172.21.15.6:0/791207178 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922540308 0 0) 0x7f90bc0b9660 con 0x7f90bc0ced00 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.143+0000 7f90c0f14640 1 -- 172.21.15.6:0/791207178 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90bc0b9900 con 0x7f90bc0d3210 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90c0f14640 1 --2- 172.21.15.6:0/791207178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0d3210 0x7f90bc0d3c30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0d23f0 tx=0x7f90b402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e052d45b3cc539 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90baffd640 1 -- 172.21.15.6:0/791207178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4039070 con 0x7f90bc0d3210 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90baffd640 1 -- 172.21.15.6:0/791207178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b402cbb0 con 0x7f90bc0d3210 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90baffd640 1 -- 172.21.15.6:0/791207178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4034440 con 0x7f90bc0d3210 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90c2977640 1 -- 172.21.15.6:0/791207178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0d3210 msgr2=0x7f90bc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90c2977640 1 --2- 172.21.15.6:0/791207178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0d3210 0x7f90bc0d3c30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0d23f0 tx=0x7f90b402be00 comp rx=0 tx=0).stop 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90c2977640 1 -- 172.21.15.6:0/791207178 shutdown_connections 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90c2977640 1 --2- 172.21.15.6:0/791207178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0d3210 0x7f90bc0d3c30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90c2977640 1 --2- 172.21.15.6:0/791207178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc15e020 0x7f90bc162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.144+0000 7f90c2977640 1 -- 172.21.15.6:0/791207178 >> 172.21.15.6:0/791207178 conn(0x7f90bc072ce0 msgr2=0x7f90bc0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.145+0000 7f90c2977640 1 -- 172.21.15.6:0/791207178 shutdown_connections 2024-03-18T14:41:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.145+0000 7f90c2977640 1 -- 172.21.15.6:0/791207178 wait complete. 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 Processor -- start 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 -- start start 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0ced00 0x7f90bc15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc0d3210 0x7f90bc1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90bc15e020 0x7f90bc160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc0cf2c0 con 0x7f90bc0ced00 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc0ce890 con 0x7f90bc15e020 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.146+0000 7f90c2977640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f90bc0cf440 con 0x7f90bc0d3210 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0ced00 0x7f90bc15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0ced00 0x7f90bc15ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48398/0 (socket says 172.21.15.6:48398) 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 -- 172.21.15.6:0/165868413 learned_addr learned my addr 172.21.15.6:0/165868413 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:47.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bb7fe640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc0d3210 0x7f90bc1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90c0f14640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90bc15e020 0x7f90bc160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 -- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc0d3210 msgr2=0x7f90bc1522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc0d3210 0x7f90bc1522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 -- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90bc15e020 msgr2=0x7f90bc160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90bc15e020 0x7f90bc160130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 -- 172.21.15.6:0/165868413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90bc160670 con 0x7f90bc0ced00 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90bbfff640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0ced00 0x7f90bc15ee10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00e8a0 tx=0x7f90ac00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90b8ff9640 1 -- 172.21.15.6:0/165868413 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac017070 con 0x7f90bc0ced00 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90b8ff9640 1 -- 172.21.15.6:0/165868413 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ac00c940 con 0x7f90bc0ced00 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90bc15f510 con 0x7f90bc0ced00 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90b8ff9640 1 -- 172.21.15.6:0/165868413 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00cd30 con 0x7f90bc0ced00 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.147+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90bc1545b0 con 0x7f90bc0ced00 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.148+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90bc08afe0 con 0x7f90bc0ced00 2024-03-18T14:41:47.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.149+0000 7f90b8ff9640 1 -- 172.21.15.6:0/165868413 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f90ac01a070 con 0x7f90bc0ced00 2024-03-18T14:41:47.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.149+0000 7f90b8ff9640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f90a806bc00 0x7f90a806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.149+0000 7f90b8ff9640 1 -- 172.21.15.6:0/165868413 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f90ac08aaf0 con 0x7f90bc0ced00 2024-03-18T14:41:47.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.149+0000 7f90bb7fe640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f90a806bc00 0x7f90a806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.150+0000 7f90bb7fe640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f90a806bc00 0x7f90a806e0c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0b7a50 tx=0x7f90b0007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.154+0000 7f90b8ff9640 1 -- 172.21.15.6:0/165868413 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f90bc160670 con 0x7f90bc0ced00 2024-03-18T14:41:47.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.335+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90bc0ca600 con 0x7f90bc0ced00 2024-03-18T14:41:47.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.336+0000 7f90b8ff9640 1 -- 172.21.15.6:0/165868413 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f90ac058ce0 con 0x7f90bc0ced00 2024-03-18T14:41:47.338 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:47.338 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f90a806bc00 msgr2=0x7f90a806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f90a806bc00 0x7f90a806e0c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0b7a50 tx=0x7f90b0007d50 comp rx=0 tx=0).stop 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0ced00 msgr2=0x7f90bc15ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0ced00 0x7f90bc15ee10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00e8a0 tx=0x7f90ac00ed60 comp rx=0 tx=0).stop 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 shutdown_connections 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f90a806bc00 0x7f90a806e0c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90bc15e020 0x7f90bc160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90bc0d3210 0x7f90bc1522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 --2- 172.21.15.6:0/165868413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90bc0ced00 0x7f90bc15ee10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.338+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 >> 172.21.15.6:0/165868413 conn(0x7f90bc072ce0 msgr2=0x7f90bc0cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.339+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 shutdown_connections 2024-03-18T14:41:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.339+0000 7f90c2977640 1 -- 172.21.15.6:0/165868413 wait complete. 2024-03-18T14:41:47.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:41:47.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:41:47.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.454+0000 7f8aa546c640 1 Processor -- start 2024-03-18T14:41:47.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.454+0000 7f8aa546c640 1 -- start start 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8aa546c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa015dfe0 0x7f8aa00d37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8aa546c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8aa00b5ce0 0x7f8aa00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8aa546c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aa00b6670 con 0x7f8aa0161990 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8aa546c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f8aa00b7450 con 0x7f8aa00b5ce0 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8aa546c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa00d07c0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9e7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8aa0161990 0x7f8aa00b6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50538/0 (socket says 172.21.15.6:50538) 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9e7fc640 1 -- 172.21.15.6:0/2210706406 learned_addr learned my addr 172.21.15.6:0/2210706406 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9f7fe640 1 --2- 172.21.15.6:0/2210706406 >> v2:172.21.15.92:3300/0 conn(0x7f8aa00b5ce0 0x7f8aa00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9effd640 1 --2- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa015dfe0 0x7f8aa00d37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9dffb640 1 -- 172.21.15.6:0/2210706406 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1885432947 0 0) 0x7f8aa00b6670 con 0x7f8aa0161990 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9dffb640 1 -- 172.21.15.6:0/2210706406 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a8c003520 con 0x7f8aa0161990 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9f7fe640 1 -- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa015dfe0 msgr2=0x7f8aa00d37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9f7fe640 1 --2- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa015dfe0 0x7f8aa00d37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9f7fe640 1 -- 172.21.15.6:0/2210706406 >> v1:172.21.15.6:6789/0 conn(0x7f8aa0161990 legacy=0x7f8aa00b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9f7fe640 1 -- 172.21.15.6:0/2210706406 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa00d3d10 con 0x7f8aa00b5ce0 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.455+0000 7f8a9effd640 1 --2- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa015dfe0 0x7f8aa00d37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.456+0000 7f8a9f7fe640 1 --2- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa00b5ce0 0x7f8aa00b6130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a980097a0 tx=0x7f8a9802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba8a1c091847c3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.456+0000 7f8a9dffb640 1 -- 172.21.15.6:0/2210706406 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98035020 con 0x7f8aa00b5ce0 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.456+0000 7f8a9dffb640 1 -- 172.21.15.6:0/2210706406 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a98031830 con 0x7f8aa00b5ce0 2024-03-18T14:41:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.456+0000 7f8a9dffb640 1 -- 172.21.15.6:0/2210706406 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98031c00 con 0x7f8aa00b5ce0 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 -- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa00b5ce0 msgr2=0x7f8aa00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 --2- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa00b5ce0 0x7f8aa00b6130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a980097a0 tx=0x7f8a9802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 -- 172.21.15.6:0/2210706406 shutdown_connections 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 --2- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa00b5ce0 0x7f8aa00b6130 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 --2- 172.21.15.6:0/2210706406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa015dfe0 0x7f8aa00d37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 -- 172.21.15.6:0/2210706406 >> 172.21.15.6:0/2210706406 conn(0x7f8aa0072ad0 msgr2=0x7f8aa0072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 -- 172.21.15.6:0/2210706406 shutdown_connections 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.457+0000 7f8aa546c640 1 -- 172.21.15.6:0/2210706406 wait complete. 2024-03-18T14:41:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.458+0000 7f8aa546c640 1 Processor -- start 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.458+0000 7f8aa546c640 1 -- start start 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8aa546c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa00b5ce0 0x7f8aa01f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8aa546c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa015dfe0 0x7f8aa00cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8aa546c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa0161990 0x7f8aa0155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8aa546c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01624c0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8aa546c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01621e0 con 0x7f8aa0161990 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8aa546c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa01627a0 con 0x7f8aa00b5ce0 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa015dfe0 0x7f8aa00cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa015dfe0 0x7f8aa00cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48410/0 (socket says 172.21.15.6:48410) 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 -- 172.21.15.6:0/3496534415 learned_addr learned my addr 172.21.15.6:0/3496534415 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9f7fe640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa0161990 0x7f8aa0155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9effd640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa00b5ce0 0x7f8aa01f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 -- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa00b5ce0 msgr2=0x7f8aa01f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa00b5ce0 0x7f8aa01f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 -- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa0161990 msgr2=0x7f8aa0155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa0161990 0x7f8aa0155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 -- 172.21.15.6:0/3496534415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa00c7fc0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.459+0000 7f8a9e7fc640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa015dfe0 0x7f8aa00cd3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c390 tx=0x7f8a8800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.460+0000 7f8a977fe640 1 -- 172.21.15.6:0/3496534415 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88017070 con 0x7f8aa015dfe0 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.460+0000 7f8a977fe640 1 -- 172.21.15.6:0/3496534415 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a88005ce0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.460+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa015e470 con 0x7f8aa015dfe0 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.460+0000 7f8a977fe640 1 -- 172.21.15.6:0/3496534415 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a880072f0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.460+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa015ea10 con 0x7f8aa015dfe0 2024-03-18T14:41:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.461+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa00751b0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.461+0000 7f8a977fe640 1 -- 172.21.15.6:0/3496534415 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8a8801a070 con 0x7f8aa015dfe0 2024-03-18T14:41:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.461+0000 7f8a977fe640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8a8006bc00 0x7f8a8006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.462+0000 7f8a977fe640 1 -- 172.21.15.6:0/3496534415 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8a8808a0c0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.462+0000 7f8a9effd640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8a8006bc00 0x7f8a8006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.462+0000 7f8a9effd640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8a8006bc00 0x7f8a8006e0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009bd0 tx=0x7f8a9000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.466+0000 7f8a977fe640 1 -- 172.21.15.6:0/3496534415 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8a88058220 con 0x7f8aa015dfe0 2024-03-18T14:41:47.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.651+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8aa006dae0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.652+0000 7f8a977fe640 1 -- 172.21.15.6:0/3496534415 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f8a8805c2c0 con 0x7f8aa015dfe0 2024-03-18T14:41:47.654 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:41:47.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8a8006bc00 msgr2=0x7f8a8006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8a8006bc00 0x7f8a8006e0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009bd0 tx=0x7f8a9000b040 comp rx=0 tx=0).stop 2024-03-18T14:41:47.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa015dfe0 msgr2=0x7f8aa00cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa015dfe0 0x7f8aa00cd3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c390 tx=0x7f8a8800c850 comp rx=0 tx=0).stop 2024-03-18T14:41:47.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 shutdown_connections 2024-03-18T14:41:47.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8a8006bc00 0x7f8a8006e0c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aa0161990 0x7f8aa0155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa015dfe0 0x7f8aa00cd3b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 --2- 172.21.15.6:0/3496534415 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aa00b5ce0 0x7f8aa01f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 >> 172.21.15.6:0/3496534415 conn(0x7f8aa0072ad0 msgr2=0x7f8aa015d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:47.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 shutdown_connections 2024-03-18T14:41:47.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.654+0000 7f8aa546c640 1 -- 172.21.15.6:0/3496534415 wait complete. 2024-03-18T14:41:47.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:41:47.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.769+0000 7ff2ea805640 1 Processor -- start 2024-03-18T14:41:47.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.769+0000 7ff2ea805640 1 -- start start 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2ea805640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff2e415dfd0 0x7ff2e40b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2ea805640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff2e40b7e00 0x7ff2e40b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2ea805640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e415e3b0 con 0x7ff2e40b5740 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2ea805640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff2e40b8240 con 0x7ff2e40b7e00 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2ea805640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff2e40b8470 con 0x7ff2e415dfd0 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e8da2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff2e40b7e00 0x7ff2e40b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff2e40b5740 0x7ff2e40d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50552/0 (socket says 172.21.15.6:50552) 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e8da2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff2e40b7e00 0x7ff2e40b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48418/0 (socket says 172.21.15.6:48418) 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e37fe640 1 -- 172.21.15.6:0/1704418208 learned_addr learned my addr 172.21.15.6:0/1704418208 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:47.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e3fff640 1 --2- 172.21.15.6:0/1704418208 >> v2:172.21.15.92:3300/0 conn(0x7ff2e415dfd0 0x7ff2e40b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e8da2640 1 -- 172.21.15.6:0/1704418208 >> v2:172.21.15.92:3300/0 conn(0x7ff2e415dfd0 msgr2=0x7ff2e40b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e8da2640 1 --2- 172.21.15.6:0/1704418208 >> v2:172.21.15.92:3300/0 conn(0x7ff2e415dfd0 0x7ff2e40b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e8da2640 1 -- 172.21.15.6:0/1704418208 >> v1:172.21.15.6:6789/0 conn(0x7ff2e40b5740 legacy=0x7ff2e40d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e8da2640 1 -- 172.21.15.6:0/1704418208 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e40b8750 con 0x7ff2e40b7e00 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.770+0000 7ff2e2ffd640 1 -- 172.21.15.6:0/1704418208 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1890578484 0 0) 0x7ff2e415e3b0 con 0x7ff2e40b5740 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.771+0000 7ff2e8da2640 1 --2- 172.21.15.6:0/1704418208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b7e00 0x7ff2e40b5bd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40d17c0 tx=0x7ff2dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf16d5df4c47a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.771+0000 7ff2e2ffd640 1 -- 172.21.15.6:0/1704418208 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc039070 con 0x7ff2e40b7e00 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.771+0000 7ff2e2ffd640 1 -- 172.21.15.6:0/1704418208 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2dc02cc80 con 0x7ff2e40b7e00 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.771+0000 7ff2e2ffd640 1 -- 172.21.15.6:0/1704418208 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc034660 con 0x7ff2e40b7e00 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 -- 172.21.15.6:0/1704418208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b7e00 msgr2=0x7ff2e40b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1704418208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b7e00 0x7ff2e40b5bd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40d17c0 tx=0x7ff2dc02bed0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 -- 172.21.15.6:0/1704418208 shutdown_connections 2024-03-18T14:41:47.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1704418208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b7e00 0x7ff2e40b5bd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1704418208 >> v2:172.21.15.92:3300/0 conn(0x7ff2e415dfd0 0x7ff2e40b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 -- 172.21.15.6:0/1704418208 >> 172.21.15.6:0/1704418208 conn(0x7ff2e4072d10 msgr2=0x7ff2e4073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:47.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 -- 172.21.15.6:0/1704418208 shutdown_connections 2024-03-18T14:41:47.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.772+0000 7ff2ea805640 1 -- 172.21.15.6:0/1704418208 wait complete. 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.773+0000 7ff2ea805640 1 Processor -- start 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.773+0000 7ff2ea805640 1 -- start start 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2ea805640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b5740 0x7ff2e415b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2ea805640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2e40b7e00 0x7ff2e40c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2ea805640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e415dfd0 0x7ff2e40c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2ea805640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e40b9b90 con 0x7ff2e40b5740 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2ea805640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e40b8d10 con 0x7ff2e415dfd0 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2ea805640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e40d2550 con 0x7ff2e40b7e00 2024-03-18T14:41:47.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b5740 0x7ff2e415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b5740 0x7ff2e415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48424/0 (socket says 172.21.15.6:48424) 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 -- 172.21.15.6:0/1878095991 learned_addr learned my addr 172.21.15.6:0/1878095991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e8da2640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e415dfd0 0x7ff2e40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e37fe640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2e40b7e00 0x7ff2e40c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 -- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2e40b7e00 msgr2=0x7ff2e40c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2e40b7e00 0x7ff2e40c9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 -- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e415dfd0 msgr2=0x7ff2e40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e415dfd0 0x7ff2e40c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 -- 172.21.15.6:0/1878095991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e41607e0 con 0x7ff2e40b5740 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e3fff640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b5740 0x7ff2e415b450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400e870 tx=0x7ff2d400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.774+0000 7ff2e0ff9640 1 -- 172.21.15.6:0/1878095991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4017070 con 0x7ff2e40b5740 2024-03-18T14:41:47.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.775+0000 7ff2e0ff9640 1 -- 172.21.15.6:0/1878095991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d400c940 con 0x7ff2e40b5740 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.775+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e4160ac0 con 0x7ff2e40b5740 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.775+0000 7ff2e0ff9640 1 -- 172.21.15.6:0/1878095991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400cd30 con 0x7ff2e40b5740 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.775+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2e4161080 con 0x7ff2e40b5740 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.775+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e40751b0 con 0x7ff2e40b5740 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.776+0000 7ff2e0ff9640 1 -- 172.21.15.6:0/1878095991 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff2d401a070 con 0x7ff2e40b5740 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.776+0000 7ff2e0ff9640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2c406bbd0 0x7ff2c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.776+0000 7ff2e0ff9640 1 -- 172.21.15.6:0/1878095991 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff2d408afc0 con 0x7ff2e40b5740 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.776+0000 7ff2e37fe640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2c406bbd0 0x7ff2c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:47.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.777+0000 7ff2e37fe640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2c406bbd0 0x7ff2c406e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0011e0 tx=0x7ff2cc001130 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:47.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.781+0000 7ff2e0ff9640 1 -- 172.21.15.6:0/1878095991 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff2d4059110 con 0x7ff2e40b5740 2024-03-18T14:41:47.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.961+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff2e40d0d80 con 0x7ff2e40b5740 2024-03-18T14:41:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.961+0000 7ff2e0ff9640 1 -- 172.21.15.6:0/1878095991 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7ff2d405d1b0 con 0x7ff2e40b5740 2024-03-18T14:41:47.963 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:41:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2c406bbd0 msgr2=0x7ff2c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2c406bbd0 0x7ff2c406e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0011e0 tx=0x7ff2cc001130 comp rx=0 tx=0).stop 2024-03-18T14:41:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b5740 msgr2=0x7ff2e415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b5740 0x7ff2e415b450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400e870 tx=0x7ff2d400ec70 comp rx=0 tx=0).stop 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 shutdown_connections 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2c406bbd0 0x7ff2c406e090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e415dfd0 0x7ff2e40c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2e40b7e00 0x7ff2e40c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 --2- 172.21.15.6:0/1878095991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2e40b5740 0x7ff2e415b450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 >> 172.21.15.6:0/1878095991 conn(0x7ff2e4072d10 msgr2=0x7ff2e40cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 shutdown_connections 2024-03-18T14:41:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:47.963+0000 7ff2ea805640 1 -- 172.21.15.6:0/1878095991 wait complete. 2024-03-18T14:41:47.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:41:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.078+0000 7efcaabd4640 1 Processor -- start 2024-03-18T14:41:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.078+0000 7efcaabd4640 1 -- start start 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efcaabd4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efca4161a00 0x7efca40b3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efcaabd4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca40b2fb0 con 0x7efca415ddb0 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efcaabd4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca4161fc0 con 0x7efca40b6400 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efcaabd4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efca40cf3f0 con 0x7efca4161a00 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efca9171640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efca4161a00 0x7efca40b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efca8970640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efca415ddb0 0x7efca40b5cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50558/0 (socket says 172.21.15.6:50558) 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efca8970640 1 -- 172.21.15.6:0/1857260991 learned_addr learned my addr 172.21.15.6:0/1857260991 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efca9171640 1 -- 172.21.15.6:0/1857260991 >> v1:172.21.15.92:6789/0 conn(0x7efca40b6400 legacy=0x7efca40d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efca9171640 1 -- 172.21.15.6:0/1857260991 >> v1:172.21.15.6:6789/0 conn(0x7efca415ddb0 legacy=0x7efca40b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efca9171640 1 -- 172.21.15.6:0/1857260991 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca41622a0 con 0x7efca4161a00 2024-03-18T14:41:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.079+0000 7efca37fe640 1 -- 172.21.15.6:0/1857260991 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635524083 0 0) 0x7efca40b2fb0 con 0x7efca415ddb0 2024-03-18T14:41:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.080+0000 7efca9171640 1 --2- 172.21.15.6:0/1857260991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca4161a00 0x7efca40b3400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0097a0 tx=0x7efc8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbb0b6b8eed679f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.080+0000 7efca37fe640 1 -- 172.21.15.6:0/1857260991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c035020 con 0x7efca4161a00 2024-03-18T14:41:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.080+0000 7efca37fe640 1 -- 172.21.15.6:0/1857260991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc8c031830 con 0x7efca4161a00 2024-03-18T14:41:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.080+0000 7efca37fe640 1 -- 172.21.15.6:0/1857260991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c031c00 con 0x7efca4161a00 2024-03-18T14:41:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.081+0000 7efcaabd4640 1 -- 172.21.15.6:0/1857260991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca4161a00 msgr2=0x7efca40b3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.081+0000 7efcaabd4640 1 --2- 172.21.15.6:0/1857260991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca4161a00 0x7efca40b3400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0097a0 tx=0x7efc8c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.081+0000 7efcaabd4640 1 -- 172.21.15.6:0/1857260991 shutdown_connections 2024-03-18T14:41:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.081+0000 7efcaabd4640 1 --2- 172.21.15.6:0/1857260991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca4161a00 0x7efca40b3400 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.081+0000 7efcaabd4640 1 -- 172.21.15.6:0/1857260991 >> 172.21.15.6:0/1857260991 conn(0x7efca4072ad0 msgr2=0x7efca4072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.081+0000 7efcaabd4640 1 -- 172.21.15.6:0/1857260991 shutdown_connections 2024-03-18T14:41:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.081+0000 7efcaabd4640 1 -- 172.21.15.6:0/1857260991 wait complete. 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.082+0000 7efcaabd4640 1 Processor -- start 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.082+0000 7efcaabd4640 1 -- start start 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efcaabd4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca40b6400 0x7efca40cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efcaabd4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efca415ddb0 0x7efca40c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efcaabd4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efca4161a00 0x7efca41f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efcaabd4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efca415e190 con 0x7efca40b6400 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efcaabd4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efca4162860 con 0x7efca415ddb0 2024-03-18T14:41:48.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efcaabd4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efca40d26f0 con 0x7efca4161a00 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca40b6400 0x7efca40cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca40b6400 0x7efca40cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48436/0 (socket says 172.21.15.6:48436) 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 -- 172.21.15.6:0/366665384 learned_addr learned my addr 172.21.15.6:0/366665384 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca3fff640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efca415ddb0 0x7efca40c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca9171640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efca4161a00 0x7efca41f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 -- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efca4161a00 msgr2=0x7efca41f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efca4161a00 0x7efca41f7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 -- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efca415ddb0 msgr2=0x7efca40c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efca415ddb0 0x7efca40c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 -- 172.21.15.6:0/366665384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca40ca500 con 0x7efca40b6400 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca9171640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efca4161a00 0x7efca41f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca8970640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca40b6400 0x7efca40cd8c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c4d0 tx=0x7efc9c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.083+0000 7efca17fa640 1 -- 172.21.15.6:0/366665384 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c018070 con 0x7efca40b6400 2024-03-18T14:41:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.084+0000 7efca17fa640 1 -- 172.21.15.6:0/366665384 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc9c0074e0 con 0x7efca40b6400 2024-03-18T14:41:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.084+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca40c7b50 con 0x7efca40b6400 2024-03-18T14:41:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.084+0000 7efca17fa640 1 -- 172.21.15.6:0/366665384 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c004a20 con 0x7efca40b6400 2024-03-18T14:41:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.084+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca40c8110 con 0x7efca40b6400 2024-03-18T14:41:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.085+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca40751b0 con 0x7efca40b6400 2024-03-18T14:41:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.085+0000 7efca17fa640 1 -- 172.21.15.6:0/366665384 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efc9c01b070 con 0x7efca40b6400 2024-03-18T14:41:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.085+0000 7efca17fa640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc8406bbd0 0x7efc8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.085+0000 7efca17fa640 1 -- 172.21.15.6:0/366665384 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efc9c089f60 con 0x7efca40b6400 2024-03-18T14:41:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.085+0000 7efca3fff640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc8406bbd0 0x7efc8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.086+0000 7efca3fff640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc8406bbd0 0x7efc8406e090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efca40c8f10 tx=0x7efc94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.090+0000 7efca17fa640 1 -- 172.21.15.6:0/366665384 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efc9c058150 con 0x7efca40b6400 2024-03-18T14:41:48.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.273+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efca40d21d0 con 0x7efca40b6400 2024-03-18T14:41:48.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.273+0000 7efca17fa640 1 -- 172.21.15.6:0/366665384 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7efc9c05c1f0 con 0x7efca40b6400 2024-03-18T14:41:48.275 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:41:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc8406bbd0 msgr2=0x7efc8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc8406bbd0 0x7efc8406e090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efca40c8f10 tx=0x7efc94009040 comp rx=0 tx=0).stop 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca40b6400 msgr2=0x7efca40cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca40b6400 0x7efca40cd8c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c4d0 tx=0x7efc9c00c990 comp rx=0 tx=0).stop 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 shutdown_connections 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc8406bbd0 0x7efc8406e090 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efca4161a00 0x7efca41f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efca415ddb0 0x7efca40c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 --2- 172.21.15.6:0/366665384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efca40b6400 0x7efca40cd8c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 >> 172.21.15.6:0/366665384 conn(0x7efca4072ad0 msgr2=0x7efca40b6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 shutdown_connections 2024-03-18T14:41:48.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.275+0000 7efcaabd4640 1 -- 172.21.15.6:0/366665384 wait complete. 2024-03-18T14:41:48.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:41:48.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.391+0000 7fe30383d640 1 Processor -- start 2024-03-18T14:41:48.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.391+0000 7fe30383d640 1 -- start start 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe30383d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe2fc0ce9d0 0x7fe2fc0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe30383d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d0730 0x7fe2fc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe30383d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc0d29a0 con 0x7fe2fc0d2510 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe30383d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe2fc0cd8d0 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe30383d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe2fc0cfac0 con 0x7fe2fc0d0730 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe300dd8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2fc0d2510 0x7fe2fc0d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50574/0 (socket says 172.21.15.6:50574) 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe300dd8640 1 -- 172.21.15.6:0/24734144 learned_addr learned my addr 172.21.15.6:0/24734144 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe3015d9640 1 --2- 172.21.15.6:0/24734144 >> v2:172.21.15.92:3300/0 conn(0x7fe2fc0ce9d0 0x7fe2fc0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe301dda640 1 --2- 172.21.15.6:0/24734144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d0730 0x7fe2fc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe2fbfff640 1 -- 172.21.15.6:0/24734144 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3619225492 0 0) 0x7fe2fc0d29a0 con 0x7fe2fc0d2510 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe2fbfff640 1 -- 172.21.15.6:0/24734144 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2e4003520 con 0x7fe2fc0d2510 2024-03-18T14:41:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe301dda640 1 -- 172.21.15.6:0/24734144 >> v2:172.21.15.92:3300/0 conn(0x7fe2fc0ce9d0 msgr2=0x7fe2fc0cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe301dda640 1 --2- 172.21.15.6:0/24734144 >> v2:172.21.15.92:3300/0 conn(0x7fe2fc0ce9d0 0x7fe2fc0cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe301dda640 1 -- 172.21.15.6:0/24734144 >> v1:172.21.15.6:6789/0 conn(0x7fe2fc0d2510 legacy=0x7fe2fc0d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe301dda640 1 -- 172.21.15.6:0/24734144 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc0cccc0 con 0x7fe2fc0d0730 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.392+0000 7fe3015d9640 1 --2- 172.21.15.6:0/24734144 >> v2:172.21.15.92:3300/0 conn(0x7fe2fc0ce9d0 0x7fe2fc0cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.393+0000 7fe301dda640 1 --2- 172.21.15.6:0/24734144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d0730 0x7fe2fc15b690 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009d90 tx=0x7fe2ec02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b32e5f3686232e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.393+0000 7fe2fbfff640 1 -- 172.21.15.6:0/24734144 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec039070 con 0x7fe2fc0d0730 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.393+0000 7fe2fbfff640 1 -- 172.21.15.6:0/24734144 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2ec02cbc0 con 0x7fe2fc0d0730 2024-03-18T14:41:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.393+0000 7fe2fbfff640 1 -- 172.21.15.6:0/24734144 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec034440 con 0x7fe2fc0d0730 2024-03-18T14:41:48.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 -- 172.21.15.6:0/24734144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d0730 msgr2=0x7fe2fc15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 --2- 172.21.15.6:0/24734144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d0730 0x7fe2fc15b690 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009d90 tx=0x7fe2ec02be10 comp rx=0 tx=0).stop 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 -- 172.21.15.6:0/24734144 shutdown_connections 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 --2- 172.21.15.6:0/24734144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d0730 0x7fe2fc15b690 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 --2- 172.21.15.6:0/24734144 >> v2:172.21.15.92:3300/0 conn(0x7fe2fc0ce9d0 0x7fe2fc0cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 -- 172.21.15.6:0/24734144 >> 172.21.15.6:0/24734144 conn(0x7fe2fc08b0c0 msgr2=0x7fe2fc0d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 -- 172.21.15.6:0/24734144 shutdown_connections 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.394+0000 7fe30383d640 1 -- 172.21.15.6:0/24734144 wait complete. 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.395+0000 7fe30383d640 1 Processor -- start 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.395+0000 7fe30383d640 1 -- start start 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe30383d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2fc0ce9d0 0x7fe2fc06dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe30383d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2fc0d0730 0x7fe2fc155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe30383d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d2510 0x7fe2fc1f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe30383d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc0cd560 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe30383d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc0cd280 con 0x7fe2fc0d0730 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe30383d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe2fc0cedb0 con 0x7fe2fc0d2510 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2fc0ce9d0 0x7fe2fc06dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2fc0ce9d0 0x7fe2fc06dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48440/0 (socket says 172.21.15.6:48440) 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 -- 172.21.15.6:0/671531922 learned_addr learned my addr 172.21.15.6:0/671531922 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe300dd8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2fc0d0730 0x7fe2fc155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe301dda640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d2510 0x7fe2fc1f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 -- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d2510 msgr2=0x7fe2fc1f7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d2510 0x7fe2fc1f7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 -- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2fc0d0730 msgr2=0x7fe2fc155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2fc0d0730 0x7fe2fc155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 -- 172.21.15.6:0/671531922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc1fba30 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe3015d9640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2fc0ce9d0 0x7fe2fc06dee0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400e9b0 tx=0x7fe2f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe2f9ffb640 1 -- 172.21.15.6:0/671531922 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4005990 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe2f9ffb640 1 -- 172.21.15.6:0/671531922 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f4005b30 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2fc1fbd10 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.396+0000 7fe2f9ffb640 1 -- 172.21.15.6:0/671531922 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400c9a0 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.397+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2fc1fc2d0 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.398+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2fc0751b0 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.398+0000 7fe2f9ffb640 1 -- 172.21.15.6:0/671531922 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe2f400cb40 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.398+0000 7fe2f9ffb640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe2e806bc00 0x7fe2e806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.398+0000 7fe2f9ffb640 1 -- 172.21.15.6:0/671531922 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe2f408b010 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.399+0000 7fe300dd8640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe2e806bc00 0x7fe2e806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.399+0000 7fe300dd8640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe2e806bc00 0x7fe2e806e0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0c84d0 tx=0x7fe2f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.403+0000 7fe2f9ffb640 1 -- 172.21.15.6:0/671531922 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe2f4017020 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.586+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe2fc0cfe00 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.587+0000 7fe2f9ffb640 1 -- 172.21.15.6:0/671531922 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fe2f4059200 con 0x7fe2fc0ce9d0 2024-03-18T14:41:48.588 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe2e806bc00 msgr2=0x7fe2e806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe2e806bc00 0x7fe2e806e0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0c84d0 tx=0x7fe2f0009040 comp rx=0 tx=0).stop 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2fc0ce9d0 msgr2=0x7fe2fc06dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2fc0ce9d0 0x7fe2fc06dee0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400e9b0 tx=0x7fe2f400ee70 comp rx=0 tx=0).stop 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 shutdown_connections 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe2e806bc00 0x7fe2e806e0c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe2fc0d2510 0x7fe2fc1f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2fc0d0730 0x7fe2fc155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 --2- 172.21.15.6:0/671531922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2fc0ce9d0 0x7fe2fc06dee0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 >> 172.21.15.6:0/671531922 conn(0x7fe2fc08b0c0 msgr2=0x7fe2fc0b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:48.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 shutdown_connections 2024-03-18T14:41:48.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.589+0000 7fe30383d640 1 -- 172.21.15.6:0/671531922 wait complete. 2024-03-18T14:41:48.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:41:48.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.702+0000 7fd3d8434640 1 Processor -- start 2024-03-18T14:41:48.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.702+0000 7fd3d8434640 1 -- start start 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d8434640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3d015bcb0 0x7fd3d00ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d8434640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d00d1bf0 0x7fd3d00d2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d8434640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d00aeef0 con 0x7fd3d015c090 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d8434640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d00d25c0 con 0x7fd3d015bcb0 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d8434640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d00afb00 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d61d0640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3d015bcb0 0x7fd3d00ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d61d0640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3d015bcb0 0x7fd3d00ae910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48444/0 (socket says 172.21.15.6:48444) 2024-03-18T14:41:48.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d61d0640 1 -- 172.21.15.6:0/2666254110 learned_addr learned my addr 172.21.15.6:0/2666254110 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:48.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d69d1640 1 --2- 172.21.15.6:0/2666254110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d00d1bf0 0x7fd3d00d2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d51ce640 1 -- 172.21.15.6:0/2666254110 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2033077124 0 0) 0x7fd3d00aeef0 con 0x7fd3d015c090 2024-03-18T14:41:48.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d61d0640 1 -- 172.21.15.6:0/2666254110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d00d1bf0 msgr2=0x7fd3d00d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:48.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d61d0640 1 --2- 172.21.15.6:0/2666254110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d00d1bf0 0x7fd3d00d2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d61d0640 1 -- 172.21.15.6:0/2666254110 >> v1:172.21.15.92:6789/0 conn(0x7fd3d015c090 legacy=0x7fd3d00d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.703+0000 7fd3d61d0640 1 -- 172.21.15.6:0/2666254110 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d00af1d0 con 0x7fd3d015bcb0 2024-03-18T14:41:48.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.704+0000 7fd3d61d0640 1 --2- 172.21.15.6:0/2666254110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d015bcb0 0x7fd3d00ae910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009870 tx=0x7fd3cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb5335343a7f62f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.704+0000 7fd3d51ce640 1 -- 172.21.15.6:0/2666254110 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc034070 con 0x7fd3d015bcb0 2024-03-18T14:41:48.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.704+0000 7fd3d51ce640 1 -- 172.21.15.6:0/2666254110 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc030840 con 0x7fd3d015bcb0 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.704+0000 7fd3d51ce640 1 -- 172.21.15.6:0/2666254110 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc030c10 con 0x7fd3d015bcb0 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.704+0000 7fd3d8434640 1 -- 172.21.15.6:0/2666254110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d015bcb0 msgr2=0x7fd3d00ae910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.704+0000 7fd3d8434640 1 --2- 172.21.15.6:0/2666254110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d015bcb0 0x7fd3d00ae910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009870 tx=0x7fd3cc02b980 comp rx=0 tx=0).stop 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.705+0000 7fd3d8434640 1 -- 172.21.15.6:0/2666254110 shutdown_connections 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.705+0000 7fd3d8434640 1 --2- 172.21.15.6:0/2666254110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d00d1bf0 0x7fd3d00d2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.705+0000 7fd3d8434640 1 --2- 172.21.15.6:0/2666254110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d015bcb0 0x7fd3d00ae910 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.705+0000 7fd3d8434640 1 -- 172.21.15.6:0/2666254110 >> 172.21.15.6:0/2666254110 conn(0x7fd3d008b320 msgr2=0x7fd3d008b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.705+0000 7fd3d8434640 1 -- 172.21.15.6:0/2666254110 shutdown_connections 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.705+0000 7fd3d8434640 1 -- 172.21.15.6:0/2666254110 wait complete. 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 Processor -- start 2024-03-18T14:41:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 -- start start 2024-03-18T14:41:48.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d00d1bf0 0x7fd3d00c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d015bcb0 0x7fd3d0151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d015c090 0x7fd3d0152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d00aff90 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d00af790 con 0x7fd3d015bcb0 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.706+0000 7fd3d8434640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d00b0110 con 0x7fd3d015c090 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d00d1bf0 0x7fd3d00c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d00d1bf0 0x7fd3d00c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48452/0 (socket says 172.21.15.6:48452) 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 -- 172.21.15.6:0/706889179 learned_addr learned my addr 172.21.15.6:0/706889179 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d69d1640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d015c090 0x7fd3d0152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d59cf640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d015bcb0 0x7fd3d0151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 -- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d015c090 msgr2=0x7fd3d0152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d015c090 0x7fd3d0152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 -- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d015bcb0 msgr2=0x7fd3d0151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d015bcb0 0x7fd3d0151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 -- 172.21.15.6:0/706889179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d00b71d0 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d61d0640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d00d1bf0 0x7fd3d00c6aa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009870 tx=0x7fd3cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3c6ffd640 1 -- 172.21.15.6:0/706889179 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc048070 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3c6ffd640 1 -- 172.21.15.6:0/706889179 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc030640 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3c6ffd640 1 -- 172.21.15.6:0/706889179 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc034040 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d00b74b0 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.707+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d00b7a70 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.708+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d00d2c90 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.709+0000 7fd3c6ffd640 1 -- 172.21.15.6:0/706889179 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd3cc037070 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.709+0000 7fd3c6ffd640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3b406bbd0 0x7fd3b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.709+0000 7fd3c6ffd640 1 -- 172.21.15.6:0/706889179 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd3cc0ac930 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.709+0000 7fd3d59cf640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3b406bbd0 0x7fd3b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:48.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.710+0000 7fd3d59cf640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3b406bbd0 0x7fd3b406e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd3d01526a0 tx=0x7fd3c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:48.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.713+0000 7fd3c6ffd640 1 -- 172.21.15.6:0/706889179 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd3cc07aaa0 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.908+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd3d004f5b0 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.908+0000 7fd3c6ffd640 1 -- 172.21.15.6:0/706889179 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7fd3cc07eb40 con 0x7fd3d00d1bf0 2024-03-18T14:41:48.910 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:41:48.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3b406bbd0 msgr2=0x7fd3b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3b406bbd0 0x7fd3b406e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd3d01526a0 tx=0x7fd3c0009040 comp rx=0 tx=0).stop 2024-03-18T14:41:48.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d00d1bf0 msgr2=0x7fd3d00c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d00d1bf0 0x7fd3d00c6aa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009870 tx=0x7fd3cc004730 comp rx=0 tx=0).stop 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 shutdown_connections 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3b406bbd0 0x7fd3b406e090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3d015c090 0x7fd3d0152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3d015bcb0 0x7fd3d0151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 --2- 172.21.15.6:0/706889179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d00d1bf0 0x7fd3d00c6aa0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.910+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 >> 172.21.15.6:0/706889179 conn(0x7fd3d008b320 msgr2=0x7fd3d006fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.911+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 shutdown_connections 2024-03-18T14:41:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:48.911+0000 7fd3d8434640 1 -- 172.21.15.6:0/706889179 wait complete. 2024-03-18T14:41:48.923 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:41:48.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:48.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T14:41:48.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[44753]: fuse finished with error 0 and tester_r 0 2024-03-18T14:41:48.950 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:41:55.051 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:41:55.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:41:55.051 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T14:41:55.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:55.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.240+0000 7feb5e431640 1 Processor -- start 2024-03-18T14:41:55.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.240+0000 7feb5e431640 1 -- start start 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5e431640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb580d0d90 0x7feb580b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5e431640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb580b9ae0 con 0x7feb5815e170 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5e431640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb580cef50 con 0x7feb580b9650 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5e431640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7feb580cf230 con 0x7feb580d0d90 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5c9ce640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb580d0d90 0x7feb580b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb57fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feb5815e170 0x7feb580d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52974/0 (socket says 172.21.15.6:52974) 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb57fff640 1 -- 172.21.15.6:0/262976915 learned_addr learned my addr 172.21.15.6:0/262976915 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb56ffd640 1 -- 172.21.15.6:0/262976915 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157217245 0 0) 0x7feb580cef50 con 0x7feb580b9650 2024-03-18T14:41:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb56ffd640 1 -- 172.21.15.6:0/262976915 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb40003520 con 0x7feb580b9650 2024-03-18T14:41:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5c9ce640 1 -- 172.21.15.6:0/262976915 >> v1:172.21.15.92:6789/0 conn(0x7feb580b9650 legacy=0x7feb580d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5c9ce640 1 -- 172.21.15.6:0/262976915 >> v1:172.21.15.6:6789/0 conn(0x7feb5815e170 legacy=0x7feb580d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb56ffd640 1 -- 172.21.15.6:0/262976915 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253530346 0 0) 0x7feb580b9ae0 con 0x7feb5815e170 2024-03-18T14:41:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.241+0000 7feb5c9ce640 1 -- 172.21.15.6:0/262976915 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb580d2b30 con 0x7feb580d0d90 2024-03-18T14:41:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.242+0000 7feb5c9ce640 1 --2- 172.21.15.6:0/262976915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb580d0d90 0x7feb580b6dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feb50009d90 tx=0x7feb5002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3a7c59f68c5b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.242+0000 7feb56ffd640 1 -- 172.21.15.6:0/262976915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50039070 con 0x7feb580d0d90 2024-03-18T14:41:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.242+0000 7feb56ffd640 1 -- 172.21.15.6:0/262976915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb5002cd00 con 0x7feb580d0d90 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.242+0000 7feb56ffd640 1 -- 172.21.15.6:0/262976915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50034640 con 0x7feb580d0d90 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.243+0000 7feb5e431640 1 -- 172.21.15.6:0/262976915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb580d0d90 msgr2=0x7feb580b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.243+0000 7feb5e431640 1 --2- 172.21.15.6:0/262976915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb580d0d90 0x7feb580b6dd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feb50009d90 tx=0x7feb5002bf50 comp rx=0 tx=0).stop 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.243+0000 7feb5e431640 1 -- 172.21.15.6:0/262976915 shutdown_connections 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.243+0000 7feb5e431640 1 --2- 172.21.15.6:0/262976915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb580d0d90 0x7feb580b6dd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.243+0000 7feb5e431640 1 -- 172.21.15.6:0/262976915 >> 172.21.15.6:0/262976915 conn(0x7feb5806c900 msgr2=0x7feb5806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.243+0000 7feb5e431640 1 -- 172.21.15.6:0/262976915 shutdown_connections 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.243+0000 7feb5e431640 1 -- 172.21.15.6:0/262976915 wait complete. 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.244+0000 7feb5e431640 1 Processor -- start 2024-03-18T14:41:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.244+0000 7feb5e431640 1 -- start start 2024-03-18T14:41:55.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5e431640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feb580b9650 0x7feb5815a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5e431640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580d0d90 0x7feb58161330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5e431640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb5815e170 0x7feb580b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5e431640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feb580d2820 con 0x7feb5815e170 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5e431640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb580d3a40 con 0x7feb580d0d90 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5e431640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7feb580ceb70 con 0x7feb580b9650 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb5815e170 0x7feb580b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb57fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feb580b9650 0x7feb5815a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb5815e170 0x7feb580b8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56240/0 (socket says 172.21.15.6:56240) 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb577fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580d0d90 0x7feb58161330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 -- 172.21.15.6:0/1740329896 learned_addr learned my addr 172.21.15.6:0/1740329896 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 -- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feb580b9650 msgr2=0x7feb5815a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feb580b9650 0x7feb5815a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 -- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580d0d90 msgr2=0x7feb58161330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580d0d90 0x7feb58161330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 -- 172.21.15.6:0/1740329896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb58160290 con 0x7feb5815e170 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.245+0000 7feb5c9ce640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb5815e170 0x7feb580b8050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feb50009d90 tx=0x7feb50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.246+0000 7feb54ff9640 1 -- 172.21.15.6:0/1740329896 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50042030 con 0x7feb5815e170 2024-03-18T14:41:55.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.246+0000 7feb54ff9640 1 -- 172.21.15.6:0/1740329896 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb5003a040 con 0x7feb5815e170 2024-03-18T14:41:55.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.246+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb58160570 con 0x7feb5815e170 2024-03-18T14:41:55.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.246+0000 7feb54ff9640 1 -- 172.21.15.6:0/1740329896 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50039070 con 0x7feb5815e170 2024-03-18T14:41:55.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.246+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb58160b30 con 0x7feb5815e170 2024-03-18T14:41:55.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.247+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb5808afe0 con 0x7feb5815e170 2024-03-18T14:41:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.247+0000 7feb54ff9640 1 -- 172.21.15.6:0/1740329896 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feb50045070 con 0x7feb5815e170 2024-03-18T14:41:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.248+0000 7feb54ff9640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feb4406bbd0 0x7feb4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.248+0000 7feb54ff9640 1 -- 172.21.15.6:0/1740329896 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7feb500acc50 con 0x7feb5815e170 2024-03-18T14:41:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.248+0000 7feb57fff640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feb4406bbd0 0x7feb4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.248+0000 7feb57fff640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feb4406bbd0 0x7feb4406e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feb48000f60 tx=0x7feb48007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.252+0000 7feb54ff9640 1 -- 172.21.15.6:0/1740329896 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feb5004c070 con 0x7feb5815e170 2024-03-18T14:41:55.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.434+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feb580b2720 con 0x7feb5815e170 2024-03-18T14:41:55.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.435+0000 7feb54ff9640 1 -- 172.21.15.6:0/1740329896 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7feb5007ad10 con 0x7feb5815e170 2024-03-18T14:41:55.437 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:55.437 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:55.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feb4406bbd0 msgr2=0x7feb4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feb4406bbd0 0x7feb4406e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feb48000f60 tx=0x7feb48007e40 comp rx=0 tx=0).stop 2024-03-18T14:41:55.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb5815e170 msgr2=0x7feb580b8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb5815e170 0x7feb580b8050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feb50009d90 tx=0x7feb50004a40 comp rx=0 tx=0).stop 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 shutdown_connections 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feb4406bbd0 0x7feb4406e090 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb5815e170 0x7feb580b8050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb580d0d90 0x7feb58161330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 --2- 172.21.15.6:0/1740329896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feb580b9650 0x7feb5815a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.437+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 >> 172.21.15.6:0/1740329896 conn(0x7feb5806c900 msgr2=0x7feb580b5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.438+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 shutdown_connections 2024-03-18T14:41:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.438+0000 7feb5e431640 1 -- 172.21.15.6:0/1740329896 wait complete. 2024-03-18T14:41:55.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:55.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.550+0000 7f73b735b640 1 Processor -- start 2024-03-18T14:41:55.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.550+0000 7f73b735b640 1 -- start start 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.550+0000 7f73b735b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f73b006b8d0 0x7f73b00cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.550+0000 7f73b735b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b00cce30 con 0x7f73b015e240 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.550+0000 7f73b735b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b00b1c20 con 0x7f73b00afd00 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.550+0000 7f73b735b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f73b00cb000 con 0x7f73b006b8d0 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73b50f7640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73b015e240 0x7f73b006b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52986/0 (socket says 172.21.15.6:52986) 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73b50f7640 1 -- 172.21.15.6:0/296310809 learned_addr learned my addr 172.21.15.6:0/296310809 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73b48f6640 1 --2- 172.21.15.6:0/296310809 >> v2:172.21.15.92:3300/0 conn(0x7f73b006b8d0 0x7f73b00cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73affff640 1 -- 172.21.15.6:0/296310809 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2144148810 0 0) 0x7f73b00cce30 con 0x7f73b015e240 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73affff640 1 -- 172.21.15.6:0/296310809 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7398003520 con 0x7f73b015e240 2024-03-18T14:41:55.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73affff640 1 -- 172.21.15.6:0/296310809 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506749010 0 0) 0x7f73b00b1c20 con 0x7f73b00afd00 2024-03-18T14:41:55.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73affff640 1 -- 172.21.15.6:0/296310809 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73b00cce30 con 0x7f73b00afd00 2024-03-18T14:41:55.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73b48f6640 1 -- 172.21.15.6:0/296310809 >> v1:172.21.15.92:6789/0 conn(0x7f73b00afd00 legacy=0x7f73b00cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73b48f6640 1 -- 172.21.15.6:0/296310809 >> v1:172.21.15.6:6789/0 conn(0x7f73b015e240 legacy=0x7f73b006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.551+0000 7f73b48f6640 1 -- 172.21.15.6:0/296310809 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b00ce940 con 0x7f73b006b8d0 2024-03-18T14:41:55.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73b48f6640 1 --2- 172.21.15.6:0/296310809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 0x7f73b00cd0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f73a40097a0 tx=0x7f73a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dd39579a4a765fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73affff640 1 -- 172.21.15.6:0/296310809 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4035020 con 0x7f73b006b8d0 2024-03-18T14:41:55.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73affff640 1 -- 172.21.15.6:0/296310809 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a4031830 con 0x7f73b006b8d0 2024-03-18T14:41:55.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73affff640 1 -- 172.21.15.6:0/296310809 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4031c00 con 0x7f73b006b8d0 2024-03-18T14:41:55.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73b735b640 1 -- 172.21.15.6:0/296310809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 msgr2=0x7f73b00cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73b735b640 1 --2- 172.21.15.6:0/296310809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 0x7f73b00cd0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f73a40097a0 tx=0x7f73a402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73b735b640 1 -- 172.21.15.6:0/296310809 shutdown_connections 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73b735b640 1 --2- 172.21.15.6:0/296310809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 0x7f73b00cd0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.552+0000 7f73b735b640 1 -- 172.21.15.6:0/296310809 >> 172.21.15.6:0/296310809 conn(0x7f73b00d09b0 msgr2=0x7f73b00d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.553+0000 7f73b735b640 1 -- 172.21.15.6:0/296310809 shutdown_connections 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.553+0000 7f73b735b640 1 -- 172.21.15.6:0/296310809 wait complete. 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 Processor -- start 2024-03-18T14:41:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 -- start start 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 0x7f73b00b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73b00afd00 0x7f73b0148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73b015e240 0x7f73b00b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f73b00ce580 con 0x7f73b015e240 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f73b00b1f90 con 0x7f73b006b8d0 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.554+0000 7f73b735b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f73b00ce700 con 0x7f73b00afd00 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73b015e240 0x7f73b00b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73b015e240 0x7f73b00b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56254/0 (socket says 172.21.15.6:56254) 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 -- 172.21.15.6:0/3181382704 learned_addr learned my addr 172.21.15.6:0/3181382704 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b48f6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73b00afd00 0x7f73b0148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b50f7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 0x7f73b00b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 -- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73b00afd00 msgr2=0x7f73b0148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73b00afd00 0x7f73b0148750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 -- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 msgr2=0x7f73b00b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 0x7f73b00b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 -- 172.21.15.6:0/3181382704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b015bd90 con 0x7f73b015e240 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b58f8640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73b015e240 0x7f73b00b50f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f73a000c7a0 tx=0x7f73a000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73adffb640 1 -- 172.21.15.6:0/3181382704 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0003b70 con 0x7f73b015e240 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73adffb640 1 -- 172.21.15.6:0/3181382704 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a0003d10 con 0x7f73b015e240 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b015c070 con 0x7f73b015e240 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73adffb640 1 -- 172.21.15.6:0/3181382704 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a000aa90 con 0x7f73b015e240 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.555+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b0148c90 con 0x7f73b015e240 2024-03-18T14:41:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.556+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b009e9a0 con 0x7f73b015e240 2024-03-18T14:41:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.557+0000 7f73adffb640 1 -- 172.21.15.6:0/3181382704 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f73a000ac30 con 0x7f73b015e240 2024-03-18T14:41:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.557+0000 7f73adffb640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f739c06bbd0 0x7f739c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.557+0000 7f73adffb640 1 -- 172.21.15.6:0/3181382704 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f73a008ac70 con 0x7f73b015e240 2024-03-18T14:41:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.557+0000 7f73b50f7640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f739c06bbd0 0x7f739c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.558+0000 7f73b50f7640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f739c06bbd0 0x7f739c06e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73a8000f60 tx=0x7f73a8007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.562+0000 7f73adffb640 1 -- 172.21.15.6:0/3181382704 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f73a0058d30 con 0x7f73b015e240 2024-03-18T14:41:55.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.748+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73b004f620 con 0x7f73b015e240 2024-03-18T14:41:55.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.749+0000 7f73adffb640 1 -- 172.21.15.6:0/3181382704 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f73a005f080 con 0x7f73b015e240 2024-03-18T14:41:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:55.751 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:55.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f739c06bbd0 msgr2=0x7f739c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f739c06bbd0 0x7f739c06e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73a8000f60 tx=0x7f73a8007d30 comp rx=0 tx=0).stop 2024-03-18T14:41:55.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73b015e240 msgr2=0x7f73b00b50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73b015e240 0x7f73b00b50f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f73a000c7a0 tx=0x7f73a000cc60 comp rx=0 tx=0).stop 2024-03-18T14:41:55.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 shutdown_connections 2024-03-18T14:41:55.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f739c06bbd0 0x7f739c06e090 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73b015e240 0x7f73b00b50f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73b00afd00 0x7f73b0148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 --2- 172.21.15.6:0/3181382704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73b006b8d0 0x7f73b00b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 >> 172.21.15.6:0/3181382704 conn(0x7f73b00d09b0 msgr2=0x7f73b006eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:55.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 shutdown_connections 2024-03-18T14:41:55.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.751+0000 7f73b735b640 1 -- 172.21.15.6:0/3181382704 wait complete. 2024-03-18T14:41:55.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T14:41:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.867+0000 7ff0cdf5b640 1 Processor -- start 2024-03-18T14:41:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.867+0000 7ff0cdf5b640 1 -- start start 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0cdf5b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff0c815dfd0 0x7ff0c80b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0cdf5b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c80b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0cdf5b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c815e3b0 con 0x7ff0c80b5740 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0cdf5b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff0c80b8470 con 0x7ff0c815dfd0 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0cdf5b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff0c80b8750 con 0x7ff0c80b7e00 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c6ffd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff0c80b5740 0x7ff0c80d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45690/0 (socket says 172.21.15.6:45690) 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c6ffd640 1 -- 172.21.15.6:0/591678198 learned_addr learned my addr 172.21.15.6:0/591678198 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c77fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff0c815dfd0 0x7ff0c80b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c7fff640 1 --2- 172.21.15.6:0/591678198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c80b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c67fc640 1 -- 172.21.15.6:0/591678198 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634718501 0 0) 0x7ff0c815e3b0 con 0x7ff0c80b5740 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c67fc640 1 -- 172.21.15.6:0/591678198 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b0003520 con 0x7ff0c80b5740 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c7fff640 1 -- 172.21.15.6:0/591678198 >> v2:172.21.15.92:3300/0 conn(0x7ff0c815dfd0 msgr2=0x7ff0c80b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c7fff640 1 --2- 172.21.15.6:0/591678198 >> v2:172.21.15.92:3300/0 conn(0x7ff0c815dfd0 0x7ff0c80b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c7fff640 1 -- 172.21.15.6:0/591678198 >> v1:172.21.15.92:6789/0 conn(0x7ff0c80b5740 legacy=0x7ff0c80d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c7fff640 1 -- 172.21.15.6:0/591678198 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c80b8a30 con 0x7ff0c80b7e00 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.868+0000 7ff0c77fe640 1 --2- 172.21.15.6:0/591678198 >> v2:172.21.15.92:3300/0 conn(0x7ff0c815dfd0 0x7ff0c80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.869+0000 7ff0c7fff640 1 --2- 172.21.15.6:0/591678198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c80b5bd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009d90 tx=0x7ff0c002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9f7cb492eaa7dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.869+0000 7ff0c67fc640 1 -- 172.21.15.6:0/591678198 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0039070 con 0x7ff0c80b7e00 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.869+0000 7ff0c67fc640 1 -- 172.21.15.6:0/591678198 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c002cc50 con 0x7ff0c80b7e00 2024-03-18T14:41:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.869+0000 7ff0c67fc640 1 -- 172.21.15.6:0/591678198 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0034440 con 0x7ff0c80b7e00 2024-03-18T14:41:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/591678198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 msgr2=0x7ff0c80b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/591678198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c80b5bd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009d90 tx=0x7ff0c002bea0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/591678198 shutdown_connections 2024-03-18T14:41:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/591678198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c80b5bd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/591678198 >> v2:172.21.15.92:3300/0 conn(0x7ff0c815dfd0 0x7ff0c80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/591678198 >> 172.21.15.6:0/591678198 conn(0x7ff0c8072d10 msgr2=0x7ff0c8073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/591678198 shutdown_connections 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.870+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/591678198 wait complete. 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.871+0000 7ff0cdf5b640 1 Processor -- start 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.871+0000 7ff0cdf5b640 1 -- start start 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0cdf5b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0c80b5740 0x7ff0c815fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0cdf5b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c81f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0cdf5b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0c815dfd0 0x7ff0c81f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0cdf5b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c80d2550 con 0x7ff0c80b5740 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0cdf5b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c80b9b90 con 0x7ff0c815dfd0 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0cdf5b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff0c80d26d0 con 0x7ff0c80b7e00 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0c80b5740 0x7ff0c815fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0c80b5740 0x7ff0c815fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56256/0 (socket says 172.21.15.6:56256) 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c7fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0c815dfd0 0x7ff0c81f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c77fe640 1 -- 172.21.15.6:0/690704372 learned_addr learned my addr 172.21.15.6:0/690704372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c6ffd640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c81f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c7fff640 1 -- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 msgr2=0x7ff0c81f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c7fff640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c81f79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c7fff640 1 -- 172.21.15.6:0/690704372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0c80b5740 msgr2=0x7ff0c815fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c7fff640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0c80b5740 0x7ff0c815fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.872+0000 7ff0c7fff640 1 -- 172.21.15.6:0/690704372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c81f4920 con 0x7ff0c815dfd0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.873+0000 7ff0c7fff640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0c815dfd0 0x7ff0c81f43e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0033000 tx=0x7ff0c0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.873+0000 7ff0affff640 1 -- 172.21.15.6:0/690704372 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0039050 con 0x7ff0c815dfd0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.873+0000 7ff0affff640 1 -- 172.21.15.6:0/690704372 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c0045070 con 0x7ff0c815dfd0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.873+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c81602f0 con 0x7ff0c815dfd0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.873+0000 7ff0affff640 1 -- 172.21.15.6:0/690704372 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0009450 con 0x7ff0c815dfd0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.873+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c81f3be0 con 0x7ff0c815dfd0 2024-03-18T14:41:55.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.874+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c81f40d0 con 0x7ff0c815dfd0 2024-03-18T14:41:55.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.874+0000 7ff0affff640 1 -- 172.21.15.6:0/690704372 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff0c003e610 con 0x7ff0c815dfd0 2024-03-18T14:41:55.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.874+0000 7ff0affff640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0b406bbd0 0x7ff0b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:55.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.875+0000 7ff0affff640 1 -- 172.21.15.6:0/690704372 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff0c00ad5d0 con 0x7ff0c815dfd0 2024-03-18T14:41:55.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.875+0000 7ff0c77fe640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0b406bbd0 0x7ff0b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:55.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.875+0000 7ff0c77fe640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0b406bbd0 0x7ff0b406e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80095e0 tx=0x7ff0b8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:55.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:55.879+0000 7ff0affff640 1 -- 172.21.15.6:0/690704372 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff0c007b840 con 0x7ff0c815dfd0 2024-03-18T14:41:56.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.099+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff0c81f39d0 con 0x7ff0c815dfd0 2024-03-18T14:41:56.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.597+0000 7ff0affff640 1 -- 172.21.15.6:0/690704372 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7ff0c007f8e0 con 0x7ff0c815dfd0 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.599+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0b406bbd0 msgr2=0x7ff0b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.599+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0b406bbd0 0x7ff0b406e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80095e0 tx=0x7ff0b8009190 comp rx=0 tx=0).stop 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.599+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0c815dfd0 msgr2=0x7ff0c81f43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.599+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0c815dfd0 0x7ff0c81f43e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0033000 tx=0x7ff0c0004a40 comp rx=0 tx=0).stop 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 shutdown_connections 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0b406bbd0 0x7ff0b406e090 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0c815dfd0 0x7ff0c81f43e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0c80b7e00 0x7ff0c81f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 --2- 172.21.15.6:0/690704372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0c80b5740 0x7ff0c815fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 >> 172.21.15.6:0/690704372 conn(0x7ff0c8072d10 msgr2=0x7ff0c80b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:56.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 shutdown_connections 2024-03-18T14:41:56.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.600+0000 7ff0cdf5b640 1 -- 172.21.15.6:0/690704372 wait complete. 2024-03-18T14:41:56.616 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-18T14:41:56.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T14:41:56.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.721+0000 7efd34187640 1 Processor -- start 2024-03-18T14:41:56.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.721+0000 7efd34187640 1 -- start start 2024-03-18T14:41:56.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd34187640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c15cf10 0x7efd2c0af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:56.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd34187640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd2c15dac0 0x7efd2c15df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:56.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd34187640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2c0afef0 con 0x7efd2c071ad0 2024-03-18T14:41:56.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd34187640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efd2c15d370 con 0x7efd2c15dac0 2024-03-18T14:41:56.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd34187640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efd2c0b01d0 con 0x7efd2c15cf10 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd32724640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd2c15dac0 0x7efd2c15df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd32724640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd2c15dac0 0x7efd2c15df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56270/0 (socket says 172.21.15.6:56270) 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd31f23640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efd2c071ad0 0x7efd2c071eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52990/0 (socket says 172.21.15.6:52990) 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd32724640 1 -- 172.21.15.6:0/479164904 learned_addr learned my addr 172.21.15.6:0/479164904 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd31722640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c15cf10 0x7efd2c0af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd32724640 1 -- 172.21.15.6:0/479164904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c15cf10 msgr2=0x7efd2c0af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd32724640 1 --2- 172.21.15.6:0/479164904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c15cf10 0x7efd2c0af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd32724640 1 -- 172.21.15.6:0/479164904 >> v1:172.21.15.6:6789/0 conn(0x7efd2c071ad0 legacy=0x7efd2c071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd32724640 1 -- 172.21.15.6:0/479164904 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c0b04b0 con 0x7efd2c15dac0 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd30f21640 1 -- 172.21.15.6:0/479164904 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1808866836 0 0) 0x7efd2c0afef0 con 0x7efd2c071ad0 2024-03-18T14:41:56.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.722+0000 7efd31722640 1 --2- 172.21.15.6:0/479164904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c15cf10 0x7efd2c0af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:56.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.723+0000 7efd32724640 1 --2- 172.21.15.6:0/479164904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15dac0 0x7efd2c15df20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efd24009e50 tx=0x7efd2402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f798978aa84f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:56.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd30f21640 1 -- 172.21.15.6:0/479164904 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2402cb00 con 0x7efd2c15dac0 2024-03-18T14:41:56.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd30f21640 1 -- 172.21.15.6:0/479164904 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd2402cca0 con 0x7efd2c15dac0 2024-03-18T14:41:56.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd30f21640 1 -- 172.21.15.6:0/479164904 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24034610 con 0x7efd2c15dac0 2024-03-18T14:41:56.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 -- 172.21.15.6:0/479164904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15dac0 msgr2=0x7efd2c15df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 --2- 172.21.15.6:0/479164904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15dac0 0x7efd2c15df20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efd24009e50 tx=0x7efd2402bf60 comp rx=0 tx=0).stop 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 -- 172.21.15.6:0/479164904 shutdown_connections 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 --2- 172.21.15.6:0/479164904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15dac0 0x7efd2c15df20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 --2- 172.21.15.6:0/479164904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c15cf10 0x7efd2c0af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 -- 172.21.15.6:0/479164904 >> 172.21.15.6:0/479164904 conn(0x7efd2c06b1c0 msgr2=0x7efd2c06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 -- 172.21.15.6:0/479164904 shutdown_connections 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.724+0000 7efd34187640 1 -- 172.21.15.6:0/479164904 wait complete. 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.725+0000 7efd34187640 1 Processor -- start 2024-03-18T14:41:56.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd34187640 1 -- start start 2024-03-18T14:41:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd34187640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c071ad0 0x7efd2c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd34187640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15cf10 0x7efd2c15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:56.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd34187640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c15dac0 0x7efd2c15b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd34187640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c159be0 con 0x7efd2c15cf10 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd34187640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c159a60 con 0x7efd2c15dac0 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd34187640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efd2c0afac0 con 0x7efd2c071ad0 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd31722640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15cf10 0x7efd2c15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd31722640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15cf10 0x7efd2c15b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56274/0 (socket says 172.21.15.6:56274) 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd31722640 1 -- 172.21.15.6:0/199710956 learned_addr learned my addr 172.21.15.6:0/199710956 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd32724640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c15dac0 0x7efd2c15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.726+0000 7efd31f23640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c071ad0 0x7efd2c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd31722640 1 -- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c071ad0 msgr2=0x7efd2c157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd31722640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c071ad0 0x7efd2c157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd31722640 1 -- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c15dac0 msgr2=0x7efd2c15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:56.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd31722640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c15dac0 0x7efd2c15b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd31722640 1 -- 172.21.15.6:0/199710956 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c0b3b50 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd31722640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15cf10 0x7efd2c15b0e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd1c00e960 tx=0x7efd1c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd227fc640 1 -- 172.21.15.6:0/199710956 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c017070 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd227fc640 1 -- 172.21.15.6:0/199710956 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c00ca00 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2c0abbe0 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd227fc640 1 -- 172.21.15.6:0/199710956 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c00cdd0 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.727+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2c0ac1a0 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.728+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2c0751b0 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.729+0000 7efd227fc640 1 -- 172.21.15.6:0/199710956 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efd1c01a070 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.729+0000 7efd227fc640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd1006bc00 0x7efd1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.729+0000 7efd227fc640 1 -- 172.21.15.6:0/199710956 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efd1c08af00 con 0x7efd2c15cf10 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.729+0000 7efd31f23640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd1006bc00 0x7efd1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:56.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.730+0000 7efd31f23640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd1006bc00 0x7efd1006e0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd28007fd0 tx=0x7efd28007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:56.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.733+0000 7efd227fc640 1 -- 172.21.15.6:0/199710956 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efd1c059070 con 0x7efd2c15cf10 2024-03-18T14:41:56.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:56.957+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7efd2c06ee10 con 0x7efd2c15cf10 2024-03-18T14:41:57.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.599+0000 7efd227fc640 1 -- 172.21.15.6:0/199710956 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7efd1c05d110 con 0x7efd2c15cf10 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd1006bc00 msgr2=0x7efd1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd1006bc00 0x7efd1006e0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd28007fd0 tx=0x7efd28007de0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15cf10 msgr2=0x7efd2c15b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15cf10 0x7efd2c15b0e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd1c00e960 tx=0x7efd1c00ee20 comp rx=0 tx=0).stop 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 shutdown_connections 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd1006bc00 0x7efd1006e0c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c15dac0 0x7efd2c15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c15cf10 0x7efd2c15b0e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 --2- 172.21.15.6:0/199710956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c071ad0 0x7efd2c157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 >> 172.21.15.6:0/199710956 conn(0x7efd2c06b1c0 msgr2=0x7efd2c06f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:57.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 shutdown_connections 2024-03-18T14:41:57.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.601+0000 7efd34187640 1 -- 172.21.15.6:0/199710956 wait complete. 2024-03-18T14:41:57.614 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T14:41:57.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:41:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.715+0000 7f5aba977640 1 Processor -- start 2024-03-18T14:41:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.715+0000 7f5aba977640 1 -- start start 2024-03-18T14:41:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5aba977640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5ab415ab10 0x7f5ab40b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5aba977640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab40cdb60 con 0x7f5ab40ae840 2024-03-18T14:41:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5aba977640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab406f120 con 0x7f5ab40cb680 2024-03-18T14:41:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5aba977640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5ab406fca0 con 0x7f5ab415ab10 2024-03-18T14:41:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5ab3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5ab415ab10 0x7f5ab40b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:57.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5ab37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ab40ae840 0x7f5ab40cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52996/0 (socket says 172.21.15.6:52996) 2024-03-18T14:41:57.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5ab3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5ab415ab10 0x7f5ab40b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56288/0 (socket says 172.21.15.6:56288) 2024-03-18T14:41:57.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.716+0000 7f5ab37fe640 1 -- 172.21.15.6:0/498192715 learned_addr learned my addr 172.21.15.6:0/498192715 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:57.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab3fff640 1 -- 172.21.15.6:0/498192715 >> v1:172.21.15.92:6789/0 conn(0x7f5ab40cb680 legacy=0x7f5ab40cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab3fff640 1 -- 172.21.15.6:0/498192715 >> v1:172.21.15.6:6789/0 conn(0x7f5ab40ae840 legacy=0x7f5ab40cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab2ffd640 1 -- 172.21.15.6:0/498192715 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650801567 0 0) 0x7f5ab406f120 con 0x7f5ab40cb680 2024-03-18T14:41:57.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab3fff640 1 -- 172.21.15.6:0/498192715 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab406f400 con 0x7f5ab415ab10 2024-03-18T14:41:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab3fff640 1 --2- 172.21.15.6:0/498192715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab415ab10 0x7f5ab40b51b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800ba40 tx=0x7f5aa802df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e076665f789fecd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab2ffd640 1 -- 172.21.15.6:0/498192715 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa803b070 con 0x7f5ab415ab10 2024-03-18T14:41:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab2ffd640 1 -- 172.21.15.6:0/498192715 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa802ec90 con 0x7f5ab415ab10 2024-03-18T14:41:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.717+0000 7f5ab2ffd640 1 -- 172.21.15.6:0/498192715 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8036660 con 0x7f5ab415ab10 2024-03-18T14:41:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.718+0000 7f5aba977640 1 -- 172.21.15.6:0/498192715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab415ab10 msgr2=0x7f5ab40b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.718+0000 7f5aba977640 1 --2- 172.21.15.6:0/498192715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab415ab10 0x7f5ab40b51b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800ba40 tx=0x7f5aa802df60 comp rx=0 tx=0).stop 2024-03-18T14:41:57.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.718+0000 7f5aba977640 1 -- 172.21.15.6:0/498192715 shutdown_connections 2024-03-18T14:41:57.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.718+0000 7f5aba977640 1 --2- 172.21.15.6:0/498192715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab415ab10 0x7f5ab40b51b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.718+0000 7f5aba977640 1 -- 172.21.15.6:0/498192715 >> 172.21.15.6:0/498192715 conn(0x7f5ab40b1470 msgr2=0x7f5ab40b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:57.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.718+0000 7f5aba977640 1 -- 172.21.15.6:0/498192715 shutdown_connections 2024-03-18T14:41:57.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.718+0000 7f5aba977640 1 -- 172.21.15.6:0/498192715 wait complete. 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.719+0000 7f5aba977640 1 Processor -- start 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.719+0000 7f5aba977640 1 -- start start 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5aba977640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab40ae840 0x7f5ab4143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5aba977640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab40cb680 0x7f5ab4154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5aba977640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ab415ab10 0x7f5ab41551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5aba977640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab415aef0 con 0x7f5ab40ae840 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5aba977640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab406f9c0 con 0x7f5ab40cb680 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5aba977640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab40c6d10 con 0x7f5ab415ab10 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab40ae840 0x7f5ab4143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab40ae840 0x7f5ab4143470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56290/0 (socket says 172.21.15.6:56290) 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5ab3fff640 1 -- 172.21.15.6:0/905341999 learned_addr learned my addr 172.21.15.6:0/905341999 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:57.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5ab8f14640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ab415ab10 0x7f5ab41551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5ab37fe640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab40cb680 0x7f5ab4154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.720+0000 7f5ab3fff640 1 -- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ab415ab10 msgr2=0x7f5ab41551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab3fff640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ab415ab10 0x7f5ab41551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab3fff640 1 -- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab40cb680 msgr2=0x7f5ab4154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab3fff640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab40cb680 0x7f5ab4154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab3fff640 1 -- 172.21.15.6:0/905341999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab415c3b0 con 0x7f5ab40ae840 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab3fff640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab40ae840 0x7f5ab4143470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8039040 tx=0x7f5aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab0ff9640 1 -- 172.21.15.6:0/905341999 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8046020 con 0x7f5ab40ae840 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab0ff9640 1 -- 172.21.15.6:0/905341999 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa8041690 con 0x7f5ab40ae840 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab415c690 con 0x7f5ab40ae840 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5ab0ff9640 1 -- 172.21.15.6:0/905341999 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa803b040 con 0x7f5ab40ae840 2024-03-18T14:41:57.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.721+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab40c6950 con 0x7f5ab40ae840 2024-03-18T14:41:57.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.722+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab4073310 con 0x7f5ab40ae840 2024-03-18T14:41:57.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.722+0000 7f5ab0ff9640 1 -- 172.21.15.6:0/905341999 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5aa8049020 con 0x7f5ab40ae840 2024-03-18T14:41:57.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.723+0000 7f5ab0ff9640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5aa006bbd0 0x7f5aa006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:57.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.723+0000 7f5ab0ff9640 1 -- 172.21.15.6:0/905341999 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5aa80af7b0 con 0x7f5ab40ae840 2024-03-18T14:41:57.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.723+0000 7f5ab37fe640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5aa006bbd0 0x7f5aa006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:57.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.723+0000 7f5ab37fe640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5aa006bbd0 0x7f5aa006e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40ce5b0 tx=0x7f5aa4007e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:57.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.727+0000 7f5ab0ff9640 1 -- 172.21.15.6:0/905341999 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5aa8050030 con 0x7f5ab40ae840 2024-03-18T14:41:57.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.948+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ab404f620 con 0x7f5ab40ae840 2024-03-18T14:41:57.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.949+0000 7f5ab0ff9640 1 -- 172.21.15.6:0/905341999 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7f5aa807d9a0 con 0x7f5ab40ae840 2024-03-18T14:41:57.951 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:57.951 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:41:57.951 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5aa006bbd0 msgr2=0x7f5aa006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5aa006bbd0 0x7f5aa006e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40ce5b0 tx=0x7f5aa4007e20 comp rx=0 tx=0).stop 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab40ae840 msgr2=0x7f5ab4143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab40ae840 0x7f5ab4143470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8039040 tx=0x7f5aa80046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 shutdown_connections 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5aa006bbd0 0x7f5aa006e090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ab415ab10 0x7f5ab41551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab40cb680 0x7f5ab4154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 --2- 172.21.15.6:0/905341999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ab40ae840 0x7f5ab4143470 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:57.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 >> 172.21.15.6:0/905341999 conn(0x7f5ab40b1470 msgr2=0x7f5ab40b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:57.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 shutdown_connections 2024-03-18T14:41:57.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:57.951+0000 7f5aba977640 1 -- 172.21.15.6:0/905341999 wait complete. 2024-03-18T14:41:57.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.063+0000 7fc3f444b640 1 Processor -- start 2024-03-18T14:41:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.063+0000 7fc3f444b640 1 -- start start 2024-03-18T14:41:58.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f444b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc3ec0b9650 0x7fc3ec0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f444b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3ec0d0bc0 0x7fc3ec0b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f444b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3ec0d0ff0 con 0x7fc3ec15e170 2024-03-18T14:41:58.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f444b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc3ec0b9ae0 con 0x7fc3ec0d0bc0 2024-03-18T14:41:58.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f444b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc3ec0cef50 con 0x7fc3ec0b9650 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f21e7640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc3ec15e170 0x7fc3ec0d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53010/0 (socket says 172.21.15.6:53010) 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f21e7640 1 -- 172.21.15.6:0/2911383057 learned_addr learned my addr 172.21.15.6:0/2911383057 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f29e8640 1 --2- 172.21.15.6:0/2911383057 >> v2:172.21.15.6:3300/0 conn(0x7fc3ec0d0bc0 0x7fc3ec0b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f19e6640 1 --2- 172.21.15.6:0/2911383057 >> v2:172.21.15.92:3300/0 conn(0x7fc3ec0b9650 0x7fc3ec0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f29e8640 1 -- 172.21.15.6:0/2911383057 >> v2:172.21.15.92:3300/0 conn(0x7fc3ec0b9650 msgr2=0x7fc3ec0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f29e8640 1 --2- 172.21.15.6:0/2911383057 >> v2:172.21.15.92:3300/0 conn(0x7fc3ec0b9650 0x7fc3ec0d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f29e8640 1 -- 172.21.15.6:0/2911383057 >> v1:172.21.15.6:6789/0 conn(0x7fc3ec15e170 legacy=0x7fc3ec0d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f11e5640 1 -- 172.21.15.6:0/2911383057 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430037465 0 0) 0x7fc3ec0d0ff0 con 0x7fc3ec15e170 2024-03-18T14:41:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.064+0000 7fc3f29e8640 1 -- 172.21.15.6:0/2911383057 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec0cf230 con 0x7fc3ec0d0bc0 2024-03-18T14:41:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.065+0000 7fc3f29e8640 1 --2- 172.21.15.6:0/2911383057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec0d0bc0 0x7fc3ec0b6dd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0d1740 tx=0x7fc3e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35096ceca7773b90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.065+0000 7fc3f11e5640 1 -- 172.21.15.6:0/2911383057 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4039070 con 0x7fc3ec0d0bc0 2024-03-18T14:41:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.065+0000 7fc3f11e5640 1 -- 172.21.15.6:0/2911383057 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e402cc80 con 0x7fc3ec0d0bc0 2024-03-18T14:41:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.065+0000 7fc3f11e5640 1 -- 172.21.15.6:0/2911383057 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4034660 con 0x7fc3ec0d0bc0 2024-03-18T14:41:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 -- 172.21.15.6:0/2911383057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec0d0bc0 msgr2=0x7fc3ec0b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 --2- 172.21.15.6:0/2911383057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec0d0bc0 0x7fc3ec0b6dd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0d1740 tx=0x7fc3e402bed0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 -- 172.21.15.6:0/2911383057 shutdown_connections 2024-03-18T14:41:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 --2- 172.21.15.6:0/2911383057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec0d0bc0 0x7fc3ec0b6dd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 --2- 172.21.15.6:0/2911383057 >> v2:172.21.15.92:3300/0 conn(0x7fc3ec0b9650 0x7fc3ec0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 -- 172.21.15.6:0/2911383057 >> 172.21.15.6:0/2911383057 conn(0x7fc3ec06c900 msgr2=0x7fc3ec06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 -- 172.21.15.6:0/2911383057 shutdown_connections 2024-03-18T14:41:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.066+0000 7fc3f444b640 1 -- 172.21.15.6:0/2911383057 wait complete. 2024-03-18T14:41:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.067+0000 7fc3f444b640 1 Processor -- start 2024-03-18T14:41:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.067+0000 7fc3f444b640 1 -- start start 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f444b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3ec0b9650 0x7fc3ec15b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f444b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3ec0d0bc0 0x7fc3ec1f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f444b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec15e170 0x7fc3ec149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f444b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec0d3a40 con 0x7fc3ec15e170 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f444b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec0d2e10 con 0x7fc3ec0b9650 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f444b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec0d2820 con 0x7fc3ec0d0bc0 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f29e8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec15e170 0x7fc3ec149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f19e6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3ec0d0bc0 0x7fc3ec1f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f21e7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3ec0b9650 0x7fc3ec15b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f29e8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec15e170 0x7fc3ec149f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56302/0 (socket says 172.21.15.6:56302) 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f29e8640 1 -- 172.21.15.6:0/1134748692 learned_addr learned my addr 172.21.15.6:0/1134748692 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f21e7640 1 -- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3ec0d0bc0 msgr2=0x7fc3ec1f72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f21e7640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3ec0d0bc0 0x7fc3ec1f72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f21e7640 1 -- 172.21.15.6:0/1134748692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec15e170 msgr2=0x7fc3ec149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f21e7640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec15e170 0x7fc3ec149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f21e7640 1 -- 172.21.15.6:0/1134748692 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec1f79c0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f19e6640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3ec0d0bc0 0x7fc3ec1f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.068+0000 7fc3f29e8640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec15e170 0x7fc3ec149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.069+0000 7fc3f21e7640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3ec0b9650 0x7fc3ec15b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c410 tx=0x7fc3e800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.069+0000 7fc3e2ffd640 1 -- 172.21.15.6:0/1134748692 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e80074e0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.069+0000 7fc3e2ffd640 1 -- 172.21.15.6:0/1134748692 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e80047e0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.069+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ec1f7ca0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.069+0000 7fc3e2ffd640 1 -- 172.21.15.6:0/1134748692 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8004bd0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.069+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ec14a4b0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.070+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3ec1f81c0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.070+0000 7fc3e2ffd640 1 -- 172.21.15.6:0/1134748692 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc3e8011070 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.071+0000 7fc3e2ffd640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.071+0000 7fc3e2ffd640 1 -- 172.21.15.6:0/1134748692 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc3e808a7d0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.071+0000 7fc3f19e6640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.071+0000 7fc3f19e6640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc3d006bbd0 0x7fc3d006e090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b8170 tx=0x7fc3dc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.075+0000 7fc3e2ffd640 1 -- 172.21.15.6:0/1134748692 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc3ec1f81c0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.264+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3ec08afe0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.265+0000 7fc3e2ffd640 1 -- 172.21.15.6:0/1134748692 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7fc3e80589f0 con 0x7fc3ec0b9650 2024-03-18T14:41:58.267 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:58.267 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc3d006bbd0 msgr2=0x7fc3d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc3d006bbd0 0x7fc3d006e090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b8170 tx=0x7fc3dc009190 comp rx=0 tx=0).stop 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3ec0b9650 msgr2=0x7fc3ec15b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3ec0b9650 0x7fc3ec15b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c410 tx=0x7fc3e800c8d0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 shutdown_connections 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3ec15e170 0x7fc3ec149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3ec0d0bc0 0x7fc3ec1f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 --2- 172.21.15.6:0/1134748692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3ec0b9650 0x7fc3ec15b9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 >> 172.21.15.6:0/1134748692 conn(0x7fc3ec06c900 msgr2=0x7fc3ec0b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 shutdown_connections 2024-03-18T14:41:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.267+0000 7fc3f444b640 1 -- 172.21.15.6:0/1134748692 wait complete. 2024-03-18T14:41:58.280 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T14:41:58.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:41:58.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.380+0000 7fa06904d640 1 Processor -- start 2024-03-18T14:41:58.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.380+0000 7fa06904d640 1 -- start start 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06904d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa0640b9650 0x7fa0640d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06904d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa0640d0bc0 0x7fa0640b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06904d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0640d0ff0 con 0x7fa06415e170 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06904d640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa0640b9ae0 con 0x7fa0640d0bc0 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06904d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa0640cef50 con 0x7fa0640b9650 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06359e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa0640d0bc0 0x7fa0640b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa062d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa06415e170 0x7fa0640d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53024/0 (socket says 172.21.15.6:53024) 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06359e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa0640d0bc0 0x7fa0640b6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56308/0 (socket says 172.21.15.6:56308) 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa062d9d640 1 -- 172.21.15.6:0/1948461459 learned_addr learned my addr 172.21.15.6:0/1948461459 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa05bfff640 1 --2- 172.21.15.6:0/1948461459 >> v2:172.21.15.92:3300/0 conn(0x7fa0640b9650 0x7fa0640d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06359e640 1 -- 172.21.15.6:0/1948461459 >> v2:172.21.15.92:3300/0 conn(0x7fa0640b9650 msgr2=0x7fa0640d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06359e640 1 --2- 172.21.15.6:0/1948461459 >> v2:172.21.15.92:3300/0 conn(0x7fa0640b9650 0x7fa0640d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06359e640 1 -- 172.21.15.6:0/1948461459 >> v1:172.21.15.6:6789/0 conn(0x7fa06415e170 legacy=0x7fa0640d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa06359e640 1 -- 172.21.15.6:0/1948461459 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0640cf230 con 0x7fa0640d0bc0 2024-03-18T14:41:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.381+0000 7fa05bfff640 1 --2- 172.21.15.6:0/1948461459 >> v2:172.21.15.92:3300/0 conn(0x7fa0640b9650 0x7fa0640d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:58.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.382+0000 7fa06359e640 1 --2- 172.21.15.6:0/1948461459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640d0bc0 0x7fa0640b6dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa0640d1740 tx=0x7fa05c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb30576f18e492c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.382+0000 7fa06259c640 1 -- 172.21.15.6:0/1948461459 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05c039070 con 0x7fa0640d0bc0 2024-03-18T14:41:58.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.382+0000 7fa06259c640 1 -- 172.21.15.6:0/1948461459 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa05c02cc80 con 0x7fa0640d0bc0 2024-03-18T14:41:58.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.382+0000 7fa06259c640 1 -- 172.21.15.6:0/1948461459 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05c034660 con 0x7fa0640d0bc0 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.382+0000 7fa06904d640 1 -- 172.21.15.6:0/1948461459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640d0bc0 msgr2=0x7fa0640b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.382+0000 7fa06904d640 1 --2- 172.21.15.6:0/1948461459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640d0bc0 0x7fa0640b6dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa0640d1740 tx=0x7fa05c02bed0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.382+0000 7fa06904d640 1 -- 172.21.15.6:0/1948461459 shutdown_connections 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.383+0000 7fa06904d640 1 --2- 172.21.15.6:0/1948461459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640d0bc0 0x7fa0640b6dd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.383+0000 7fa06904d640 1 --2- 172.21.15.6:0/1948461459 >> v2:172.21.15.92:3300/0 conn(0x7fa0640b9650 0x7fa0640d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.383+0000 7fa06904d640 1 -- 172.21.15.6:0/1948461459 >> 172.21.15.6:0/1948461459 conn(0x7fa06406c900 msgr2=0x7fa06406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.383+0000 7fa06904d640 1 -- 172.21.15.6:0/1948461459 shutdown_connections 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.383+0000 7fa06904d640 1 -- 172.21.15.6:0/1948461459 wait complete. 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.384+0000 7fa06904d640 1 Processor -- start 2024-03-18T14:41:58.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.384+0000 7fa06904d640 1 -- start start 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640b9650 0x7fa06414cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0640d0bc0 0x7fa0641f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06415e170 0x7fa0641f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa0640d3a40 con 0x7fa0640b9650 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa0640d2e10 con 0x7fa0640d0bc0 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa0640d2820 con 0x7fa06415e170 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640b9650 0x7fa06414cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640b9650 0x7fa06414cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56312/0 (socket says 172.21.15.6:56312) 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 -- 172.21.15.6:0/3772472080 learned_addr learned my addr 172.21.15.6:0/3772472080 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa05bfff640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0640d0bc0 0x7fa0641f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06359e640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06415e170 0x7fa0641f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 -- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06415e170 msgr2=0x7fa0641f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06415e170 0x7fa0641f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 -- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0640d0bc0 msgr2=0x7fa0641f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0640d0bc0 0x7fa0641f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 -- 172.21.15.6:0/3772472080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0640cbbf0 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa062d9d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640b9650 0x7fa06414cbc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa05400c3c0 tx=0x7fa05400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa05b7fe640 1 -- 172.21.15.6:0/3772472080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa054017070 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa05b7fe640 1 -- 172.21.15.6:0/3772472080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa054005ce0 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0640cbed0 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa05b7fe640 1 -- 172.21.15.6:0/3772472080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0540072f0 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.385+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0640cc490 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.386+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa06408afe0 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.387+0000 7fa05b7fe640 1 -- 172.21.15.6:0/3772472080 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa05401a070 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.387+0000 7fa05b7fe640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa04406bbd0 0x7fa04406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.387+0000 7fa05b7fe640 1 -- 172.21.15.6:0/3772472080 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa05408a020 con 0x7fa0640b9650 2024-03-18T14:41:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.387+0000 7fa05bfff640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa04406bbd0 0x7fa04406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.388+0000 7fa05bfff640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa04406bbd0 0x7fa04406e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009560 tx=0x7fa04c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.391+0000 7fa05b7fe640 1 -- 172.21.15.6:0/3772472080 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa054058210 con 0x7fa0640b9650 2024-03-18T14:41:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.571+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0641f3bf0 con 0x7fa04406bbd0 2024-03-18T14:41:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.587+0000 7fa05b7fe640 1 -- 172.21.15.6:0/3772472080 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fa0641f3bf0 con 0x7fa04406bbd0 2024-03-18T14:41:58.593 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa04406bbd0 msgr2=0x7fa04406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa04406bbd0 0x7fa04406e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009560 tx=0x7fa04c009190 comp rx=0 tx=0).stop 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640b9650 msgr2=0x7fa06414cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640b9650 0x7fa06414cbc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa05400c3c0 tx=0x7fa05400c880 comp rx=0 tx=0).stop 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 shutdown_connections 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa04406bbd0 0x7fa04406e090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06415e170 0x7fa0641f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0640d0bc0 0x7fa0641f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 --2- 172.21.15.6:0/3772472080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0640b9650 0x7fa06414cbc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 >> 172.21.15.6:0/3772472080 conn(0x7fa06406c900 msgr2=0x7fa0640b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 shutdown_connections 2024-03-18T14:41:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.591+0000 7fa06904d640 1 -- 172.21.15.6:0/3772472080 wait complete. 2024-03-18T14:41:58.604 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-03-18T14:41:57.384459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002613"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779658+0000","last_change":"2024-03-18T14:41:34.533975+0000","last_active":"2024-03-18T14:41:40.779658+0000","last_peered":"2024-03-18T14:41:40.779658+0000","last_clean":"2024-03-18T14:41:40.779658+0000","last_became_active":"2024-03-18T14:41:34.533697+0000","last_became_peered":"2024-03-18T14:41:34.533697+0000","last_unstale":"2024-03-18T14:41:40.779658+0000","last_undegraded":"2024-03-18T14:41:40.779658+0000","last_fullsized":"2024-03-18T14:41:40.779658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:35.555471+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:35.555193+0000","last_became_peered":"2024-03-18T14:41:35.555193+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563408+0000","last_change":"2024-03-18T14:41:39.604430+0000","last_active":"2024-03-18T14:41:40.563408+0000","last_peered":"2024-03-18T14:41:40.563408+0000","last_clean":"2024-03-18T14:41:40.563408+0000","last_became_active":"2024-03-18T14:41:39.603958+0000","last_became_peered":"2024-03-18T14:41:39.603958+0000","last_unstale":"2024-03-18T14:41:40.563408+0000","last_undegraded":"2024-03-18T14:41:40.563408+0000","last_fullsized":"2024-03-18T14:41:40.563408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:17.770515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564369+0000","last_change":"2024-03-18T14:41:34.535172+0000","last_active":"2024-03-18T14:41:40.564369+0000","last_peered":"2024-03-18T14:41:40.564369+0000","last_clean":"2024-03-18T14:41:40.564369+0000","last_became_active":"2024-03-18T14:41:34.534849+0000","last_became_peered":"2024-03-18T14:41:34.534849+0000","last_unstale":"2024-03-18T14:41:40.564369+0000","last_undegraded":"2024-03-18T14:41:40.564369+0000","last_fullsized":"2024-03-18T14:41:40.564369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779347+0000","last_change":"2024-03-18T14:41:35.553407+0000","last_active":"2024-03-18T14:41:40.779347+0000","last_peered":"2024-03-18T14:41:40.779347+0000","last_clean":"2024-03-18T14:41:40.779347+0000","last_became_active":"2024-03-18T14:41:35.553091+0000","last_became_peered":"2024-03-18T14:41:35.553091+0000","last_unstale":"2024-03-18T14:41:40.779347+0000","last_undegraded":"2024-03-18T14:41:40.779347+0000","last_fullsized":"2024-03-18T14:41:40.779347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559102+0000","last_change":"2024-03-18T14:41:39.579025+0000","last_active":"2024-03-18T14:41:40.559102+0000","last_peered":"2024-03-18T14:41:40.559102+0000","last_clean":"2024-03-18T14:41:40.559102+0000","last_became_active":"2024-03-18T14:41:39.578650+0000","last_became_peered":"2024-03-18T14:41:39.578650+0000","last_unstale":"2024-03-18T14:41:40.559102+0000","last_undegraded":"2024-03-18T14:41:40.559102+0000","last_fullsized":"2024-03-18T14:41:40.559102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:53.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778929+0000","last_change":"2024-03-18T14:41:34.536627+0000","last_active":"2024-03-18T14:41:40.778929+0000","last_peered":"2024-03-18T14:41:40.778929+0000","last_clean":"2024-03-18T14:41:40.778929+0000","last_became_active":"2024-03-18T14:41:34.536136+0000","last_became_peered":"2024-03-18T14:41:34.536136+0000","last_unstale":"2024-03-18T14:41:40.778929+0000","last_undegraded":"2024-03-18T14:41:40.778929+0000","last_fullsized":"2024-03-18T14:41:40.778929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:12.036369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559536+0000","last_change":"2024-03-18T14:41:35.544918+0000","last_active":"2024-03-18T14:41:40.559536+0000","last_peered":"2024-03-18T14:41:40.559536+0000","last_clean":"2024-03-18T14:41:40.559536+0000","last_became_active":"2024-03-18T14:41:35.544612+0000","last_became_peered":"2024-03-18T14:41:35.544612+0000","last_unstale":"2024-03-18T14:41:40.559536+0000","last_undegraded":"2024-03-18T14:41:40.559536+0000","last_fullsized":"2024-03-18T14:41:40.559536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560251+0000","last_change":"2024-03-18T14:41:39.580665+0000","last_active":"2024-03-18T14:41:40.560251+0000","last_peered":"2024-03-18T14:41:40.560251+0000","last_clean":"2024-03-18T14:41:40.560251+0000","last_became_active":"2024-03-18T14:41:39.580231+0000","last_became_peered":"2024-03-18T14:41:39.580231+0000","last_unstale":"2024-03-18T14:41:40.560251+0000","last_undegraded":"2024-03-18T14:41:40.560251+0000","last_fullsized":"2024-03-18T14:41:40.560251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778491+0000","last_change":"2024-03-18T14:41:34.531008+0000","last_active":"2024-03-18T14:41:40.778491+0000","last_peered":"2024-03-18T14:41:40.778491+0000","last_clean":"2024-03-18T14:41:40.778491+0000","last_became_active":"2024-03-18T14:41:34.530587+0000","last_became_peered":"2024-03-18T14:41:34.530587+0000","last_unstale":"2024-03-18T14:41:40.778491+0000","last_undegraded":"2024-03-18T14:41:40.778491+0000","last_fullsized":"2024-03-18T14:41:40.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560259+0000","last_change":"2024-03-18T14:41:35.542756+0000","last_active":"2024-03-18T14:41:40.560259+0000","last_peered":"2024-03-18T14:41:40.560259+0000","last_clean":"2024-03-18T14:41:40.560259+0000","last_became_active":"2024-03-18T14:41:35.542486+0000","last_became_peered":"2024-03-18T14:41:35.542486+0000","last_unstale":"2024-03-18T14:41:40.560259+0000","last_undegraded":"2024-03-18T14:41:40.560259+0000","last_fullsized":"2024-03-18T14:41:40.560259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559437+0000","last_change":"2024-03-18T14:41:39.578229+0000","last_active":"2024-03-18T14:41:40.559437+0000","last_peered":"2024-03-18T14:41:40.559437+0000","last_clean":"2024-03-18T14:41:40.559437+0000","last_became_active":"2024-03-18T14:41:39.577812+0000","last_became_peered":"2024-03-18T14:41:39.577812+0000","last_unstale":"2024-03-18T14:41:40.559437+0000","last_undegraded":"2024-03-18T14:41:40.559437+0000","last_fullsized":"2024-03-18T14:41:40.559437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:50.123586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778041+0000","last_change":"2024-03-18T14:41:35.549686+0000","last_active":"2024-03-18T14:41:40.778041+0000","last_peered":"2024-03-18T14:41:40.778041+0000","last_clean":"2024-03-18T14:41:40.778041+0000","last_became_active":"2024-03-18T14:41:35.549240+0000","last_became_peered":"2024-03-18T14:41:35.549240+0000","last_unstale":"2024-03-18T14:41:40.778041+0000","last_undegraded":"2024-03-18T14:41:40.778041+0000","last_fullsized":"2024-03-18T14:41:40.778041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778484+0000","last_change":"2024-03-18T14:41:34.530798+0000","last_active":"2024-03-18T14:41:40.778484+0000","last_peered":"2024-03-18T14:41:40.778484+0000","last_clean":"2024-03-18T14:41:40.778484+0000","last_became_active":"2024-03-18T14:41:34.530415+0000","last_became_peered":"2024-03-18T14:41:34.530415+0000","last_unstale":"2024-03-18T14:41:40.778484+0000","last_undegraded":"2024-03-18T14:41:40.778484+0000","last_fullsized":"2024-03-18T14:41:40.778484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560204+0000","last_change":"2024-03-18T14:41:39.584982+0000","last_active":"2024-03-18T14:41:40.560204+0000","last_peered":"2024-03-18T14:41:40.560204+0000","last_clean":"2024-03-18T14:41:40.560204+0000","last_became_active":"2024-03-18T14:41:39.584630+0000","last_became_peered":"2024-03-18T14:41:39.584630+0000","last_unstale":"2024-03-18T14:41:40.560204+0000","last_undegraded":"2024-03-18T14:41:40.560204+0000","last_fullsized":"2024-03-18T14:41:40.560204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563830+0000","last_change":"2024-03-18T14:41:35.552305+0000","last_active":"2024-03-18T14:41:40.563830+0000","last_peered":"2024-03-18T14:41:40.563830+0000","last_clean":"2024-03-18T14:41:40.563830+0000","last_became_active":"2024-03-18T14:41:35.552031+0000","last_became_peered":"2024-03-18T14:41:35.552031+0000","last_unstale":"2024-03-18T14:41:40.563830+0000","last_undegraded":"2024-03-18T14:41:40.563830+0000","last_fullsized":"2024-03-18T14:41:40.563830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779939+0000","last_change":"2024-03-18T14:41:34.530930+0000","last_active":"2024-03-18T14:41:40.779939+0000","last_peered":"2024-03-18T14:41:40.779939+0000","last_clean":"2024-03-18T14:41:40.779939+0000","last_became_active":"2024-03-18T14:41:34.530645+0000","last_became_peered":"2024-03-18T14:41:34.530645+0000","last_unstale":"2024-03-18T14:41:40.779939+0000","last_undegraded":"2024-03-18T14:41:40.779939+0000","last_fullsized":"2024-03-18T14:41:40.779939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559457+0000","last_change":"2024-03-18T14:41:39.585926+0000","last_active":"2024-03-18T14:41:40.559457+0000","last_peered":"2024-03-18T14:41:40.559457+0000","last_clean":"2024-03-18T14:41:40.559457+0000","last_became_active":"2024-03-18T14:41:39.585465+0000","last_became_peered":"2024-03-18T14:41:39.585465+0000","last_unstale":"2024-03-18T14:41:40.559457+0000","last_undegraded":"2024-03-18T14:41:40.559457+0000","last_fullsized":"2024-03-18T14:41:40.559457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563386+0000","last_change":"2024-03-18T14:41:35.541836+0000","last_active":"2024-03-18T14:41:40.563386+0000","last_peered":"2024-03-18T14:41:40.563386+0000","last_clean":"2024-03-18T14:41:40.563386+0000","last_became_active":"2024-03-18T14:41:35.541532+0000","last_became_peered":"2024-03-18T14:41:35.541532+0000","last_unstale":"2024-03-18T14:41:40.563386+0000","last_undegraded":"2024-03-18T14:41:40.563386+0000","last_fullsized":"2024-03-18T14:41:40.563386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778143+0000","last_change":"2024-03-18T14:41:34.529169+0000","last_active":"2024-03-18T14:41:40.778143+0000","last_peered":"2024-03-18T14:41:40.778143+0000","last_clean":"2024-03-18T14:41:40.778143+0000","last_became_active":"2024-03-18T14:41:34.526870+0000","last_became_peered":"2024-03-18T14:41:34.526870+0000","last_unstale":"2024-03-18T14:41:40.778143+0000","last_undegraded":"2024-03-18T14:41:40.778143+0000","last_fullsized":"2024-03-18T14:41:40.778143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779714+0000","last_change":"2024-03-18T14:41:39.603928+0000","last_active":"2024-03-18T14:41:40.779714+0000","last_peered":"2024-03-18T14:41:40.779714+0000","last_clean":"2024-03-18T14:41:40.779714+0000","last_became_active":"2024-03-18T14:41:39.603236+0000","last_became_peered":"2024-03-18T14:41:39.603236+0000","last_unstale":"2024-03-18T14:41:40.779714+0000","last_undegraded":"2024-03-18T14:41:40.779714+0000","last_fullsized":"2024-03-18T14:41:40.779714+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559885+0000","last_change":"2024-03-18T14:41:35.538978+0000","last_active":"2024-03-18T14:41:40.559885+0000","last_peered":"2024-03-18T14:41:40.559885+0000","last_clean":"2024-03-18T14:41:40.559885+0000","last_became_active":"2024-03-18T14:41:35.538570+0000","last_became_peered":"2024-03-18T14:41:35.538570+0000","last_unstale":"2024-03-18T14:41:40.559885+0000","last_undegraded":"2024-03-18T14:41:40.559885+0000","last_fullsized":"2024-03-18T14:41:40.559885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:38.574781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777921+0000","last_change":"2024-03-18T14:41:34.537652+0000","last_active":"2024-03-18T14:41:40.777921+0000","last_peered":"2024-03-18T14:41:40.777921+0000","last_clean":"2024-03-18T14:41:40.777921+0000","last_became_active":"2024-03-18T14:41:34.537411+0000","last_became_peered":"2024-03-18T14:41:34.537411+0000","last_unstale":"2024-03-18T14:41:40.777921+0000","last_undegraded":"2024-03-18T14:41:40.777921+0000","last_fullsized":"2024-03-18T14:41:40.777921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560190+0000","last_change":"2024-03-18T14:41:39.580821+0000","last_active":"2024-03-18T14:41:40.560190+0000","last_peered":"2024-03-18T14:41:40.560190+0000","last_clean":"2024-03-18T14:41:40.560190+0000","last_became_active":"2024-03-18T14:41:39.580317+0000","last_became_peered":"2024-03-18T14:41:39.580317+0000","last_unstale":"2024-03-18T14:41:40.560190+0000","last_undegraded":"2024-03-18T14:41:40.560190+0000","last_fullsized":"2024-03-18T14:41:40.560190+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778799+0000","last_change":"2024-03-18T14:41:35.552139+0000","last_active":"2024-03-18T14:41:40.778799+0000","last_peered":"2024-03-18T14:41:40.778799+0000","last_clean":"2024-03-18T14:41:40.778799+0000","last_became_active":"2024-03-18T14:41:35.551809+0000","last_became_peered":"2024-03-18T14:41:35.551809+0000","last_unstale":"2024-03-18T14:41:40.778799+0000","last_undegraded":"2024-03-18T14:41:40.778799+0000","last_fullsized":"2024-03-18T14:41:40.778799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779124+0000","last_change":"2024-03-18T14:41:34.534282+0000","last_active":"2024-03-18T14:41:40.779124+0000","last_peered":"2024-03-18T14:41:40.779124+0000","last_clean":"2024-03-18T14:41:40.779124+0000","last_became_active":"2024-03-18T14:41:34.533937+0000","last_became_peered":"2024-03-18T14:41:34.533937+0000","last_unstale":"2024-03-18T14:41:40.779124+0000","last_undegraded":"2024-03-18T14:41:40.779124+0000","last_fullsized":"2024-03-18T14:41:40.779124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559601+0000","last_change":"2024-03-18T14:41:39.602565+0000","last_active":"2024-03-18T14:41:40.559601+0000","last_peered":"2024-03-18T14:41:40.559601+0000","last_clean":"2024-03-18T14:41:40.559601+0000","last_became_active":"2024-03-18T14:41:39.602164+0000","last_became_peered":"2024-03-18T14:41:39.602164+0000","last_unstale":"2024-03-18T14:41:40.559601+0000","last_undegraded":"2024-03-18T14:41:40.559601+0000","last_fullsized":"2024-03-18T14:41:40.559601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564632+0000","last_change":"2024-03-18T14:41:35.547600+0000","last_active":"2024-03-18T14:41:40.564632+0000","last_peered":"2024-03-18T14:41:40.564632+0000","last_clean":"2024-03-18T14:41:40.564632+0000","last_became_active":"2024-03-18T14:41:35.547324+0000","last_became_peered":"2024-03-18T14:41:35.547324+0000","last_unstale":"2024-03-18T14:41:40.564632+0000","last_undegraded":"2024-03-18T14:41:40.564632+0000","last_fullsized":"2024-03-18T14:41:40.564632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779055+0000","last_change":"2024-03-18T14:41:34.537441+0000","last_active":"2024-03-18T14:41:40.779055+0000","last_peered":"2024-03-18T14:41:40.779055+0000","last_clean":"2024-03-18T14:41:40.779055+0000","last_became_active":"2024-03-18T14:41:34.536599+0000","last_became_peered":"2024-03-18T14:41:34.536599+0000","last_unstale":"2024-03-18T14:41:40.779055+0000","last_undegraded":"2024-03-18T14:41:40.779055+0000","last_fullsized":"2024-03-18T14:41:40.779055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563656+0000","last_change":"2024-03-18T14:41:39.604429+0000","last_active":"2024-03-18T14:41:40.563656+0000","last_peered":"2024-03-18T14:41:40.563656+0000","last_clean":"2024-03-18T14:41:40.563656+0000","last_became_active":"2024-03-18T14:41:39.604046+0000","last_became_peered":"2024-03-18T14:41:39.604046+0000","last_unstale":"2024-03-18T14:41:40.563656+0000","last_undegraded":"2024-03-18T14:41:40.563656+0000","last_fullsized":"2024-03-18T14:41:40.563656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:21.835327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778649+0000","last_change":"2024-03-18T14:41:35.539912+0000","last_active":"2024-03-18T14:41:40.778649+0000","last_peered":"2024-03-18T14:41:40.778649+0000","last_clean":"2024-03-18T14:41:40.778649+0000","last_became_active":"2024-03-18T14:41:35.538765+0000","last_became_peered":"2024-03-18T14:41:35.538765+0000","last_unstale":"2024-03-18T14:41:40.778649+0000","last_undegraded":"2024-03-18T14:41:40.778649+0000","last_fullsized":"2024-03-18T14:41:40.778649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779021+0000","last_change":"2024-03-18T14:41:34.537358+0000","last_active":"2024-03-18T14:41:40.779021+0000","last_peered":"2024-03-18T14:41:40.779021+0000","last_clean":"2024-03-18T14:41:40.779021+0000","last_became_active":"2024-03-18T14:41:34.537020+0000","last_became_peered":"2024-03-18T14:41:34.537020+0000","last_unstale":"2024-03-18T14:41:40.779021+0000","last_undegraded":"2024-03-18T14:41:40.779021+0000","last_fullsized":"2024-03-18T14:41:40.779021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:24.398074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778734+0000","last_change":"2024-03-18T14:41:39.601563+0000","last_active":"2024-03-18T14:41:40.778734+0000","last_peered":"2024-03-18T14:41:40.778734+0000","last_clean":"2024-03-18T14:41:40.778734+0000","last_became_active":"2024-03-18T14:41:39.600878+0000","last_became_peered":"2024-03-18T14:41:39.600878+0000","last_unstale":"2024-03-18T14:41:40.778734+0000","last_undegraded":"2024-03-18T14:41:40.778734+0000","last_fullsized":"2024-03-18T14:41:40.778734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:41.241691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560348+0000","last_change":"2024-03-18T14:41:35.551415+0000","last_active":"2024-03-18T14:41:40.560348+0000","last_peered":"2024-03-18T14:41:40.560348+0000","last_clean":"2024-03-18T14:41:40.560348+0000","last_became_active":"2024-03-18T14:41:35.550751+0000","last_became_peered":"2024-03-18T14:41:35.550751+0000","last_unstale":"2024-03-18T14:41:40.560348+0000","last_undegraded":"2024-03-18T14:41:40.560348+0000","last_fullsized":"2024-03-18T14:41:40.560348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778749+0000","last_change":"2024-03-18T14:41:34.534249+0000","last_active":"2024-03-18T14:41:40.778749+0000","last_peered":"2024-03-18T14:41:40.778749+0000","last_clean":"2024-03-18T14:41:40.778749+0000","last_became_active":"2024-03-18T14:41:34.533913+0000","last_became_peered":"2024-03-18T14:41:34.533913+0000","last_unstale":"2024-03-18T14:41:40.778749+0000","last_undegraded":"2024-03-18T14:41:40.778749+0000","last_fullsized":"2024-03-18T14:41:40.778749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:39.582572+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:39.581652+0000","last_became_peered":"2024-03-18T14:41:39.581652+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560327+0000","last_change":"2024-03-18T14:41:35.561297+0000","last_active":"2024-03-18T14:41:40.560327+0000","last_peered":"2024-03-18T14:41:40.560327+0000","last_clean":"2024-03-18T14:41:40.560327+0000","last_became_active":"2024-03-18T14:41:35.558687+0000","last_became_peered":"2024-03-18T14:41:35.558687+0000","last_unstale":"2024-03-18T14:41:40.560327+0000","last_undegraded":"2024-03-18T14:41:40.560327+0000","last_fullsized":"2024-03-18T14:41:40.560327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:25.399581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559780+0000","last_change":"2024-03-18T14:41:34.525754+0000","last_active":"2024-03-18T14:41:40.559780+0000","last_peered":"2024-03-18T14:41:40.559780+0000","last_clean":"2024-03-18T14:41:40.559780+0000","last_became_active":"2024-03-18T14:41:34.525453+0000","last_became_peered":"2024-03-18T14:41:34.525453+0000","last_unstale":"2024-03-18T14:41:40.559780+0000","last_undegraded":"2024-03-18T14:41:40.559780+0000","last_fullsized":"2024-03-18T14:41:40.559780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778594+0000","last_change":"2024-03-18T14:41:39.584128+0000","last_active":"2024-03-18T14:41:40.778594+0000","last_peered":"2024-03-18T14:41:40.778594+0000","last_clean":"2024-03-18T14:41:40.778594+0000","last_became_active":"2024-03-18T14:41:39.582472+0000","last_became_peered":"2024-03-18T14:41:39.582472+0000","last_unstale":"2024-03-18T14:41:40.778594+0000","last_undegraded":"2024-03-18T14:41:40.778594+0000","last_fullsized":"2024-03-18T14:41:40.778594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778049+0000","last_change":"2024-03-18T14:41:35.540356+0000","last_active":"2024-03-18T14:41:40.778049+0000","last_peered":"2024-03-18T14:41:40.778049+0000","last_clean":"2024-03-18T14:41:40.778049+0000","last_became_active":"2024-03-18T14:41:35.540060+0000","last_became_peered":"2024-03-18T14:41:35.540060+0000","last_unstale":"2024-03-18T14:41:40.778049+0000","last_undegraded":"2024-03-18T14:41:40.778049+0000","last_fullsized":"2024-03-18T14:41:40.778049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778106+0000","last_change":"2024-03-18T14:41:34.529076+0000","last_active":"2024-03-18T14:41:40.778106+0000","last_peered":"2024-03-18T14:41:40.778106+0000","last_clean":"2024-03-18T14:41:40.778106+0000","last_became_active":"2024-03-18T14:41:34.528794+0000","last_became_peered":"2024-03-18T14:41:34.528794+0000","last_unstale":"2024-03-18T14:41:40.778106+0000","last_undegraded":"2024-03-18T14:41:40.778106+0000","last_fullsized":"2024-03-18T14:41:40.778106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779855+0000","last_change":"2024-03-18T14:41:39.581385+0000","last_active":"2024-03-18T14:41:40.779855+0000","last_peered":"2024-03-18T14:41:40.779855+0000","last_clean":"2024-03-18T14:41:40.779855+0000","last_became_active":"2024-03-18T14:41:39.580972+0000","last_became_peered":"2024-03-18T14:41:39.580972+0000","last_unstale":"2024-03-18T14:41:40.779855+0000","last_undegraded":"2024-03-18T14:41:40.779855+0000","last_fullsized":"2024-03-18T14:41:40.779855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778334+0000","last_change":"2024-03-18T14:41:35.552692+0000","last_active":"2024-03-18T14:41:40.778334+0000","last_peered":"2024-03-18T14:41:40.778334+0000","last_clean":"2024-03-18T14:41:40.778334+0000","last_became_active":"2024-03-18T14:41:35.552340+0000","last_became_peered":"2024-03-18T14:41:35.552340+0000","last_unstale":"2024-03-18T14:41:40.778334+0000","last_undegraded":"2024-03-18T14:41:40.778334+0000","last_fullsized":"2024-03-18T14:41:40.778334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778015+0000","last_change":"2024-03-18T14:41:34.531973+0000","last_active":"2024-03-18T14:41:40.778015+0000","last_peered":"2024-03-18T14:41:40.778015+0000","last_clean":"2024-03-18T14:41:40.778015+0000","last_became_active":"2024-03-18T14:41:34.531796+0000","last_became_peered":"2024-03-18T14:41:34.531796+0000","last_unstale":"2024-03-18T14:41:40.778015+0000","last_undegraded":"2024-03-18T14:41:40.778015+0000","last_fullsized":"2024-03-18T14:41:40.778015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778621+0000","last_change":"2024-03-18T14:41:39.601287+0000","last_active":"2024-03-18T14:41:40.778621+0000","last_peered":"2024-03-18T14:41:40.778621+0000","last_clean":"2024-03-18T14:41:40.778621+0000","last_became_active":"2024-03-18T14:41:39.600855+0000","last_became_peered":"2024-03-18T14:41:39.600855+0000","last_unstale":"2024-03-18T14:41:40.778621+0000","last_undegraded":"2024-03-18T14:41:40.778621+0000","last_fullsized":"2024-03-18T14:41:40.778621+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563301+0000","last_change":"2024-03-18T14:41:35.547848+0000","last_active":"2024-03-18T14:41:40.563301+0000","last_peered":"2024-03-18T14:41:40.563301+0000","last_clean":"2024-03-18T14:41:40.563301+0000","last_became_active":"2024-03-18T14:41:35.547437+0000","last_became_peered":"2024-03-18T14:41:35.547437+0000","last_unstale":"2024-03-18T14:41:40.563301+0000","last_undegraded":"2024-03-18T14:41:40.563301+0000","last_fullsized":"2024-03-18T14:41:40.563301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559408+0000","last_change":"2024-03-18T14:41:34.528039+0000","last_active":"2024-03-18T14:41:40.559408+0000","last_peered":"2024-03-18T14:41:40.559408+0000","last_clean":"2024-03-18T14:41:40.559408+0000","last_became_active":"2024-03-18T14:41:34.527756+0000","last_became_peered":"2024-03-18T14:41:34.527756+0000","last_unstale":"2024-03-18T14:41:40.559408+0000","last_undegraded":"2024-03-18T14:41:40.559408+0000","last_fullsized":"2024-03-18T14:41:40.559408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778101+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:41:40.778101+0000","last_peered":"2024-03-18T14:41:40.778101+0000","last_clean":"2024-03-18T14:41:40.778101+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:41:40.778101+0000","last_undegraded":"2024-03-18T14:41:40.778101+0000","last_fullsized":"2024-03-18T14:41:40.778101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778699+0000","last_change":"2024-03-18T14:41:39.602108+0000","last_active":"2024-03-18T14:41:40.778699+0000","last_peered":"2024-03-18T14:41:40.778699+0000","last_clean":"2024-03-18T14:41:40.778699+0000","last_became_active":"2024-03-18T14:41:39.601732+0000","last_became_peered":"2024-03-18T14:41:39.601732+0000","last_unstale":"2024-03-18T14:41:40.778699+0000","last_undegraded":"2024-03-18T14:41:40.778699+0000","last_fullsized":"2024-03-18T14:41:40.778699+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778841+0000","last_change":"2024-03-18T14:41:35.544776+0000","last_active":"2024-03-18T14:41:40.778841+0000","last_peered":"2024-03-18T14:41:40.778841+0000","last_clean":"2024-03-18T14:41:40.778841+0000","last_became_active":"2024-03-18T14:41:35.544498+0000","last_became_peered":"2024-03-18T14:41:35.544498+0000","last_unstale":"2024-03-18T14:41:40.778841+0000","last_undegraded":"2024-03-18T14:41:40.778841+0000","last_fullsized":"2024-03-18T14:41:40.778841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780098+0000","last_change":"2024-03-18T14:41:34.531645+0000","last_active":"2024-03-18T14:41:40.780098+0000","last_peered":"2024-03-18T14:41:40.780098+0000","last_clean":"2024-03-18T14:41:40.780098+0000","last_became_active":"2024-03-18T14:41:34.530361+0000","last_became_peered":"2024-03-18T14:41:34.530361+0000","last_unstale":"2024-03-18T14:41:40.780098+0000","last_undegraded":"2024-03-18T14:41:40.780098+0000","last_fullsized":"2024-03-18T14:41:40.780098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778063+0000","last_change":"2024-03-18T14:41:39.601190+0000","last_active":"2024-03-18T14:41:40.778063+0000","last_peered":"2024-03-18T14:41:40.778063+0000","last_clean":"2024-03-18T14:41:40.778063+0000","last_became_active":"2024-03-18T14:41:39.600809+0000","last_became_peered":"2024-03-18T14:41:39.600809+0000","last_unstale":"2024-03-18T14:41:40.778063+0000","last_undegraded":"2024-03-18T14:41:40.778063+0000","last_fullsized":"2024-03-18T14:41:40.778063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559630+0000","last_change":"2024-03-18T14:41:35.549443+0000","last_active":"2024-03-18T14:41:40.559630+0000","last_peered":"2024-03-18T14:41:40.559630+0000","last_clean":"2024-03-18T14:41:40.559630+0000","last_became_active":"2024-03-18T14:41:35.546376+0000","last_became_peered":"2024-03-18T14:41:35.546376+0000","last_unstale":"2024-03-18T14:41:40.559630+0000","last_undegraded":"2024-03-18T14:41:40.559630+0000","last_fullsized":"2024-03-18T14:41:40.559630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779349+0000","last_change":"2024-03-18T14:41:34.528408+0000","last_active":"2024-03-18T14:41:40.779349+0000","last_peered":"2024-03-18T14:41:40.779349+0000","last_clean":"2024-03-18T14:41:40.779349+0000","last_became_active":"2024-03-18T14:41:34.527981+0000","last_became_peered":"2024-03-18T14:41:34.527981+0000","last_unstale":"2024-03-18T14:41:40.779349+0000","last_undegraded":"2024-03-18T14:41:40.779349+0000","last_fullsized":"2024-03-18T14:41:40.779349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778976+0000","last_change":"2024-03-18T14:41:39.586214+0000","last_active":"2024-03-18T14:41:40.778976+0000","last_peered":"2024-03-18T14:41:40.778976+0000","last_clean":"2024-03-18T14:41:40.778976+0000","last_became_active":"2024-03-18T14:41:39.585773+0000","last_became_peered":"2024-03-18T14:41:39.585773+0000","last_unstale":"2024-03-18T14:41:40.778976+0000","last_undegraded":"2024-03-18T14:41:40.778976+0000","last_fullsized":"2024-03-18T14:41:40.778976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:52:45.131607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778850+0000","last_change":"2024-03-18T14:41:35.553371+0000","last_active":"2024-03-18T14:41:40.778850+0000","last_peered":"2024-03-18T14:41:40.778850+0000","last_clean":"2024-03-18T14:41:40.778850+0000","last_became_active":"2024-03-18T14:41:35.552906+0000","last_became_peered":"2024-03-18T14:41:35.552906+0000","last_unstale":"2024-03-18T14:41:40.778850+0000","last_undegraded":"2024-03-18T14:41:40.778850+0000","last_fullsized":"2024-03-18T14:41:40.778850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778853+0000","last_change":"2024-03-18T14:41:34.532843+0000","last_active":"2024-03-18T14:41:40.778853+0000","last_peered":"2024-03-18T14:41:40.778853+0000","last_clean":"2024-03-18T14:41:40.778853+0000","last_became_active":"2024-03-18T14:41:34.532183+0000","last_became_peered":"2024-03-18T14:41:34.532183+0000","last_unstale":"2024-03-18T14:41:40.778853+0000","last_undegraded":"2024-03-18T14:41:40.778853+0000","last_fullsized":"2024-03-18T14:41:40.778853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559016+0000","last_change":"2024-03-18T14:41:39.603278+0000","last_active":"2024-03-18T14:41:40.559016+0000","last_peered":"2024-03-18T14:41:40.559016+0000","last_clean":"2024-03-18T14:41:40.559016+0000","last_became_active":"2024-03-18T14:41:39.602927+0000","last_became_peered":"2024-03-18T14:41:39.602927+0000","last_unstale":"2024-03-18T14:41:40.559016+0000","last_undegraded":"2024-03-18T14:41:40.559016+0000","last_fullsized":"2024-03-18T14:41:40.559016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778747+0000","last_change":"2024-03-18T14:41:35.552924+0000","last_active":"2024-03-18T14:41:40.778747+0000","last_peered":"2024-03-18T14:41:40.778747+0000","last_clean":"2024-03-18T14:41:40.778747+0000","last_became_active":"2024-03-18T14:41:35.552514+0000","last_became_peered":"2024-03-18T14:41:35.552514+0000","last_unstale":"2024-03-18T14:41:40.778747+0000","last_undegraded":"2024-03-18T14:41:40.778747+0000","last_fullsized":"2024-03-18T14:41:40.778747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560464+0000","last_change":"2024-03-18T14:41:34.534565+0000","last_active":"2024-03-18T14:41:40.560464+0000","last_peered":"2024-03-18T14:41:40.560464+0000","last_clean":"2024-03-18T14:41:40.560464+0000","last_became_active":"2024-03-18T14:41:34.534302+0000","last_became_peered":"2024-03-18T14:41:34.534302+0000","last_unstale":"2024-03-18T14:41:40.560464+0000","last_undegraded":"2024-03-18T14:41:40.560464+0000","last_fullsized":"2024-03-18T14:41:40.560464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559373+0000","last_change":"2024-03-18T14:41:39.580698+0000","last_active":"2024-03-18T14:41:40.559373+0000","last_peered":"2024-03-18T14:41:40.559373+0000","last_clean":"2024-03-18T14:41:40.559373+0000","last_became_active":"2024-03-18T14:41:39.580161+0000","last_became_peered":"2024-03-18T14:41:39.580161+0000","last_unstale":"2024-03-18T14:41:40.559373+0000","last_undegraded":"2024-03-18T14:41:40.559373+0000","last_fullsized":"2024-03-18T14:41:40.559373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:03.265239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778352+0000","last_change":"2024-03-18T14:41:35.547876+0000","last_active":"2024-03-18T14:41:40.778352+0000","last_peered":"2024-03-18T14:41:40.778352+0000","last_clean":"2024-03-18T14:41:40.778352+0000","last_became_active":"2024-03-18T14:41:35.547615+0000","last_became_peered":"2024-03-18T14:41:35.547615+0000","last_unstale":"2024-03-18T14:41:40.778352+0000","last_undegraded":"2024-03-18T14:41:40.778352+0000","last_fullsized":"2024-03-18T14:41:40.778352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777856+0000","last_change":"2024-03-18T14:41:34.526247+0000","last_active":"2024-03-18T14:41:40.777856+0000","last_peered":"2024-03-18T14:41:40.777856+0000","last_clean":"2024-03-18T14:41:40.777856+0000","last_became_active":"2024-03-18T14:41:34.525952+0000","last_became_peered":"2024-03-18T14:41:34.525952+0000","last_unstale":"2024-03-18T14:41:40.777856+0000","last_undegraded":"2024-03-18T14:41:40.777856+0000","last_fullsized":"2024-03-18T14:41:40.777856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:39.580945+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:39.580421+0000","last_became_peered":"2024-03-18T14:41:39.580421+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778016+0000","last_change":"2024-03-18T14:41:35.554425+0000","last_active":"2024-03-18T14:41:40.778016+0000","last_peered":"2024-03-18T14:41:40.778016+0000","last_clean":"2024-03-18T14:41:40.778016+0000","last_became_active":"2024-03-18T14:41:35.554164+0000","last_became_peered":"2024-03-18T14:41:35.554164+0000","last_unstale":"2024-03-18T14:41:40.778016+0000","last_undegraded":"2024-03-18T14:41:40.778016+0000","last_fullsized":"2024-03-18T14:41:40.778016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778159+0000","last_change":"2024-03-18T14:41:34.528726+0000","last_active":"2024-03-18T14:41:40.778159+0000","last_peered":"2024-03-18T14:41:40.778159+0000","last_clean":"2024-03-18T14:41:40.778159+0000","last_became_active":"2024-03-18T14:41:34.528424+0000","last_became_peered":"2024-03-18T14:41:34.528424+0000","last_unstale":"2024-03-18T14:41:40.778159+0000","last_undegraded":"2024-03-18T14:41:40.778159+0000","last_fullsized":"2024-03-18T14:41:40.778159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564629+0000","last_change":"2024-03-18T14:41:39.602221+0000","last_active":"2024-03-18T14:41:40.564629+0000","last_peered":"2024-03-18T14:41:40.564629+0000","last_clean":"2024-03-18T14:41:40.564629+0000","last_became_active":"2024-03-18T14:41:39.601652+0000","last_became_peered":"2024-03-18T14:41:39.601652+0000","last_unstale":"2024-03-18T14:41:40.564629+0000","last_undegraded":"2024-03-18T14:41:40.564629+0000","last_fullsized":"2024-03-18T14:41:40.564629+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:00.363235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778400+0000","last_change":"2024-03-18T14:41:35.548474+0000","last_active":"2024-03-18T14:41:40.778400+0000","last_peered":"2024-03-18T14:41:40.778400+0000","last_clean":"2024-03-18T14:41:40.778400+0000","last_became_active":"2024-03-18T14:41:35.548193+0000","last_became_peered":"2024-03-18T14:41:35.548193+0000","last_unstale":"2024-03-18T14:41:40.778400+0000","last_undegraded":"2024-03-18T14:41:40.778400+0000","last_fullsized":"2024-03-18T14:41:40.778400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559701+0000","last_change":"2024-03-18T14:41:34.529770+0000","last_active":"2024-03-18T14:41:40.559701+0000","last_peered":"2024-03-18T14:41:40.559701+0000","last_clean":"2024-03-18T14:41:40.559701+0000","last_became_active":"2024-03-18T14:41:34.529499+0000","last_became_peered":"2024-03-18T14:41:34.529499+0000","last_unstale":"2024-03-18T14:41:40.559701+0000","last_undegraded":"2024-03-18T14:41:40.559701+0000","last_fullsized":"2024-03-18T14:41:40.559701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564084+0000","last_change":"2024-03-18T14:41:39.601622+0000","last_active":"2024-03-18T14:41:40.564084+0000","last_peered":"2024-03-18T14:41:40.564084+0000","last_clean":"2024-03-18T14:41:40.564084+0000","last_became_active":"2024-03-18T14:41:39.601209+0000","last_became_peered":"2024-03-18T14:41:39.601209+0000","last_unstale":"2024-03-18T14:41:40.564084+0000","last_undegraded":"2024-03-18T14:41:40.564084+0000","last_fullsized":"2024-03-18T14:41:40.564084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:11.443644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778881+0000","last_change":"2024-03-18T14:41:35.547642+0000","last_active":"2024-03-18T14:41:40.778881+0000","last_peered":"2024-03-18T14:41:40.778881+0000","last_clean":"2024-03-18T14:41:40.778881+0000","last_became_active":"2024-03-18T14:41:35.547354+0000","last_became_peered":"2024-03-18T14:41:35.547354+0000","last_unstale":"2024-03-18T14:41:40.778881+0000","last_undegraded":"2024-03-18T14:41:40.778881+0000","last_fullsized":"2024-03-18T14:41:40.778881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779200+0000","last_change":"2024-03-18T14:41:34.533998+0000","last_active":"2024-03-18T14:41:40.779200+0000","last_peered":"2024-03-18T14:41:40.779200+0000","last_clean":"2024-03-18T14:41:40.779200+0000","last_became_active":"2024-03-18T14:41:34.533597+0000","last_became_peered":"2024-03-18T14:41:34.533597+0000","last_unstale":"2024-03-18T14:41:40.779200+0000","last_undegraded":"2024-03-18T14:41:40.779200+0000","last_fullsized":"2024-03-18T14:41:40.779200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560211+0000","last_change":"2024-03-18T14:41:39.601126+0000","last_active":"2024-03-18T14:41:40.560211+0000","last_peered":"2024-03-18T14:41:40.560211+0000","last_clean":"2024-03-18T14:41:40.560211+0000","last_became_active":"2024-03-18T14:41:39.600751+0000","last_became_peered":"2024-03-18T14:41:39.600751+0000","last_unstale":"2024-03-18T14:41:40.560211+0000","last_undegraded":"2024-03-18T14:41:40.560211+0000","last_fullsized":"2024-03-18T14:41:40.560211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778816+0000","last_change":"2024-03-18T14:41:35.550195+0000","last_active":"2024-03-18T14:41:40.778816+0000","last_peered":"2024-03-18T14:41:40.778816+0000","last_clean":"2024-03-18T14:41:40.778816+0000","last_became_active":"2024-03-18T14:41:35.549894+0000","last_became_peered":"2024-03-18T14:41:35.549894+0000","last_unstale":"2024-03-18T14:41:40.778816+0000","last_undegraded":"2024-03-18T14:41:40.778816+0000","last_fullsized":"2024-03-18T14:41:40.778816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:45.045404+0000","last_change":"2024-03-18T14:41:34.536841+0000","last_active":"2024-03-18T14:41:45.045404+0000","last_peered":"2024-03-18T14:41:45.045404+0000","last_clean":"2024-03-18T14:41:45.045404+0000","last_became_active":"2024-03-18T14:41:34.536536+0000","last_became_peered":"2024-03-18T14:41:34.536536+0000","last_unstale":"2024-03-18T14:41:45.045404+0000","last_undegraded":"2024-03-18T14:41:45.045404+0000","last_fullsized":"2024-03-18T14:41:45.045404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779808+0000","last_change":"2024-03-18T14:41:39.582692+0000","last_active":"2024-03-18T14:41:40.779808+0000","last_peered":"2024-03-18T14:41:40.779808+0000","last_clean":"2024-03-18T14:41:40.779808+0000","last_became_active":"2024-03-18T14:41:39.582285+0000","last_became_peered":"2024-03-18T14:41:39.582285+0000","last_unstale":"2024-03-18T14:41:40.779808+0000","last_undegraded":"2024-03-18T14:41:40.779808+0000","last_fullsized":"2024-03-18T14:41:40.779808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559959+0000","last_change":"2024-03-18T14:41:35.547692+0000","last_active":"2024-03-18T14:41:40.559959+0000","last_peered":"2024-03-18T14:41:40.559959+0000","last_clean":"2024-03-18T14:41:40.559959+0000","last_became_active":"2024-03-18T14:41:35.547381+0000","last_became_peered":"2024-03-18T14:41:35.547381+0000","last_unstale":"2024-03-18T14:41:40.559959+0000","last_undegraded":"2024-03-18T14:41:40.559959+0000","last_fullsized":"2024-03-18T14:41:40.559959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:13.037877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779251+0000","last_change":"2024-03-18T14:41:34.529622+0000","last_active":"2024-03-18T14:41:40.779251+0000","last_peered":"2024-03-18T14:41:40.779251+0000","last_clean":"2024-03-18T14:41:40.779251+0000","last_became_active":"2024-03-18T14:41:34.528752+0000","last_became_peered":"2024-03-18T14:41:34.528752+0000","last_unstale":"2024-03-18T14:41:40.779251+0000","last_undegraded":"2024-03-18T14:41:40.779251+0000","last_fullsized":"2024-03-18T14:41:40.779251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560121+0000","last_change":"2024-03-18T14:41:39.585230+0000","last_active":"2024-03-18T14:41:40.560121+0000","last_peered":"2024-03-18T14:41:40.560121+0000","last_clean":"2024-03-18T14:41:40.560121+0000","last_became_active":"2024-03-18T14:41:39.584896+0000","last_became_peered":"2024-03-18T14:41:39.584896+0000","last_unstale":"2024-03-18T14:41:40.560121+0000","last_undegraded":"2024-03-18T14:41:40.560121+0000","last_fullsized":"2024-03-18T14:41:40.560121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:35.552508+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:35.552221+0000","last_became_peered":"2024-03-18T14:41:35.552221+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560164+0000","last_change":"2024-03-18T14:41:34.534226+0000","last_active":"2024-03-18T14:41:40.560164+0000","last_peered":"2024-03-18T14:41:40.560164+0000","last_clean":"2024-03-18T14:41:40.560164+0000","last_became_active":"2024-03-18T14:41:34.533977+0000","last_became_peered":"2024-03-18T14:41:34.533977+0000","last_unstale":"2024-03-18T14:41:40.560164+0000","last_undegraded":"2024-03-18T14:41:40.560164+0000","last_fullsized":"2024-03-18T14:41:40.560164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563415+0000","last_change":"2024-03-18T14:41:39.583657+0000","last_active":"2024-03-18T14:41:40.563415+0000","last_peered":"2024-03-18T14:41:40.563415+0000","last_clean":"2024-03-18T14:41:40.563415+0000","last_became_active":"2024-03-18T14:41:39.583280+0000","last_became_peered":"2024-03-18T14:41:39.583280+0000","last_unstale":"2024-03-18T14:41:40.563415+0000","last_undegraded":"2024-03-18T14:41:40.563415+0000","last_fullsized":"2024-03-18T14:41:40.563415+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560126+0000","last_change":"2024-03-18T14:41:35.546368+0000","last_active":"2024-03-18T14:41:40.560126+0000","last_peered":"2024-03-18T14:41:40.560126+0000","last_clean":"2024-03-18T14:41:40.560126+0000","last_became_active":"2024-03-18T14:41:35.546101+0000","last_became_peered":"2024-03-18T14:41:35.546101+0000","last_unstale":"2024-03-18T14:41:40.560126+0000","last_undegraded":"2024-03-18T14:41:40.560126+0000","last_fullsized":"2024-03-18T14:41:40.560126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779013+0000","last_change":"2024-03-18T14:41:34.535621+0000","last_active":"2024-03-18T14:41:40.779013+0000","last_peered":"2024-03-18T14:41:40.779013+0000","last_clean":"2024-03-18T14:41:40.779013+0000","last_became_active":"2024-03-18T14:41:34.535351+0000","last_became_peered":"2024-03-18T14:41:34.535351+0000","last_unstale":"2024-03-18T14:41:40.779013+0000","last_undegraded":"2024-03-18T14:41:40.779013+0000","last_fullsized":"2024-03-18T14:41:40.779013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563431+0000","last_change":"2024-03-18T14:41:39.602374+0000","last_active":"2024-03-18T14:41:40.563431+0000","last_peered":"2024-03-18T14:41:40.563431+0000","last_clean":"2024-03-18T14:41:40.563431+0000","last_became_active":"2024-03-18T14:41:39.600913+0000","last_became_peered":"2024-03-18T14:41:39.600913+0000","last_unstale":"2024-03-18T14:41:40.563431+0000","last_undegraded":"2024-03-18T14:41:40.563431+0000","last_fullsized":"2024-03-18T14:41:40.563431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778290+0000","last_change":"2024-03-18T14:41:39.602659+0000","last_active":"2024-03-18T14:41:40.778290+0000","last_peered":"2024-03-18T14:41:40.778290+0000","last_clean":"2024-03-18T14:41:40.778290+0000","last_became_active":"2024-03-18T14:41:39.602286+0000","last_became_peered":"2024-03-18T14:41:39.602286+0000","last_unstale":"2024-03-18T14:41:40.778290+0000","last_undegraded":"2024-03-18T14:41:40.778290+0000","last_fullsized":"2024-03-18T14:41:40.778290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778038+0000","last_change":"2024-03-18T14:41:35.545020+0000","last_active":"2024-03-18T14:41:40.778038+0000","last_peered":"2024-03-18T14:41:40.778038+0000","last_clean":"2024-03-18T14:41:40.778038+0000","last_became_active":"2024-03-18T14:41:35.544697+0000","last_became_peered":"2024-03-18T14:41:35.544697+0000","last_unstale":"2024-03-18T14:41:40.778038+0000","last_undegraded":"2024-03-18T14:41:40.778038+0000","last_fullsized":"2024-03-18T14:41:40.778038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559154+0000","last_change":"2024-03-18T14:41:34.527624+0000","last_active":"2024-03-18T14:41:40.559154+0000","last_peered":"2024-03-18T14:41:40.559154+0000","last_clean":"2024-03-18T14:41:40.559154+0000","last_became_active":"2024-03-18T14:41:34.527291+0000","last_became_peered":"2024-03-18T14:41:34.527291+0000","last_unstale":"2024-03-18T14:41:40.559154+0000","last_undegraded":"2024-03-18T14:41:40.559154+0000","last_fullsized":"2024-03-18T14:41:40.559154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779046+0000","last_change":"2024-03-18T14:41:39.575946+0000","last_active":"2024-03-18T14:41:40.779046+0000","last_peered":"2024-03-18T14:41:40.779046+0000","last_clean":"2024-03-18T14:41:40.779046+0000","last_became_active":"2024-03-18T14:41:39.575618+0000","last_became_peered":"2024-03-18T14:41:39.575618+0000","last_unstale":"2024-03-18T14:41:40.779046+0000","last_undegraded":"2024-03-18T14:41:40.779046+0000","last_fullsized":"2024-03-18T14:41:40.779046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563258+0000","last_change":"2024-03-18T14:41:35.548573+0000","last_active":"2024-03-18T14:41:40.563258+0000","last_peered":"2024-03-18T14:41:40.563258+0000","last_clean":"2024-03-18T14:41:40.563258+0000","last_became_active":"2024-03-18T14:41:35.548284+0000","last_became_peered":"2024-03-18T14:41:35.548284+0000","last_unstale":"2024-03-18T14:41:40.563258+0000","last_undegraded":"2024-03-18T14:41:40.563258+0000","last_fullsized":"2024-03-18T14:41:40.563258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563616+0000","last_change":"2024-03-18T14:41:34.536158+0000","last_active":"2024-03-18T14:41:40.563616+0000","last_peered":"2024-03-18T14:41:40.563616+0000","last_clean":"2024-03-18T14:41:40.563616+0000","last_became_active":"2024-03-18T14:41:34.535879+0000","last_became_peered":"2024-03-18T14:41:34.535879+0000","last_unstale":"2024-03-18T14:41:40.563616+0000","last_undegraded":"2024-03-18T14:41:40.563616+0000","last_fullsized":"2024-03-18T14:41:40.563616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778365+0000","last_change":"2024-03-18T14:41:34.528073+0000","last_active":"2024-03-18T14:41:40.778365+0000","last_peered":"2024-03-18T14:41:40.778365+0000","last_clean":"2024-03-18T14:41:40.778365+0000","last_became_active":"2024-03-18T14:41:34.527245+0000","last_became_peered":"2024-03-18T14:41:34.527245+0000","last_unstale":"2024-03-18T14:41:40.778365+0000","last_undegraded":"2024-03-18T14:41:40.778365+0000","last_fullsized":"2024-03-18T14:41:40.778365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779106+0000","last_change":"2024-03-18T14:41:35.560334+0000","last_active":"2024-03-18T14:41:40.779106+0000","last_peered":"2024-03-18T14:41:40.779106+0000","last_clean":"2024-03-18T14:41:40.779106+0000","last_became_active":"2024-03-18T14:41:35.559938+0000","last_became_peered":"2024-03-18T14:41:35.559938+0000","last_unstale":"2024-03-18T14:41:40.779106+0000","last_undegraded":"2024-03-18T14:41:40.779106+0000","last_fullsized":"2024-03-18T14:41:40.779106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780010+0000","last_change":"2024-03-18T14:41:39.578410+0000","last_active":"2024-03-18T14:41:40.780010+0000","last_peered":"2024-03-18T14:41:40.780010+0000","last_clean":"2024-03-18T14:41:40.780010+0000","last_became_active":"2024-03-18T14:41:39.577919+0000","last_became_peered":"2024-03-18T14:41:39.577919+0000","last_unstale":"2024-03-18T14:41:40.780010+0000","last_undegraded":"2024-03-18T14:41:40.780010+0000","last_fullsized":"2024-03-18T14:41:40.780010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:59.569962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563940+0000","last_change":"2024-03-18T14:41:34.535233+0000","last_active":"2024-03-18T14:41:40.563940+0000","last_peered":"2024-03-18T14:41:40.563940+0000","last_clean":"2024-03-18T14:41:40.563940+0000","last_became_active":"2024-03-18T14:41:34.534879+0000","last_became_peered":"2024-03-18T14:41:34.534879+0000","last_unstale":"2024-03-18T14:41:40.563940+0000","last_undegraded":"2024-03-18T14:41:40.563940+0000","last_fullsized":"2024-03-18T14:41:40.563940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559603+0000","last_change":"2024-03-18T14:41:35.535546+0000","last_active":"2024-03-18T14:41:40.559603+0000","last_peered":"2024-03-18T14:41:40.559603+0000","last_clean":"2024-03-18T14:41:40.559603+0000","last_became_active":"2024-03-18T14:41:35.535269+0000","last_became_peered":"2024-03-18T14:41:35.535269+0000","last_unstale":"2024-03-18T14:41:40.559603+0000","last_undegraded":"2024-03-18T14:41:40.559603+0000","last_fullsized":"2024-03-18T14:41:40.559603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:39.583244+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:39.582882+0000","last_became_peered":"2024-03-18T14:41:39.582882+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:10:48.564938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777919+0000","last_change":"2024-03-18T14:41:35.554708+0000","last_active":"2024-03-18T14:41:40.777919+0000","last_peered":"2024-03-18T14:41:40.777919+0000","last_clean":"2024-03-18T14:41:40.777919+0000","last_became_active":"2024-03-18T14:41:35.554411+0000","last_became_peered":"2024-03-18T14:41:35.554411+0000","last_unstale":"2024-03-18T14:41:40.777919+0000","last_undegraded":"2024-03-18T14:41:40.777919+0000","last_fullsized":"2024-03-18T14:41:40.777919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777933+0000","last_change":"2024-03-18T14:41:35.543401+0000","last_active":"2024-03-18T14:41:40.777933+0000","last_peered":"2024-03-18T14:41:40.777933+0000","last_clean":"2024-03-18T14:41:40.777933+0000","last_became_active":"2024-03-18T14:41:35.543034+0000","last_became_peered":"2024-03-18T14:41:35.543034+0000","last_unstale":"2024-03-18T14:41:40.777933+0000","last_undegraded":"2024-03-18T14:41:40.777933+0000","last_fullsized":"2024-03-18T14:41:40.777933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559616+0000","last_change":"2024-03-18T14:41:35.551162+0000","last_active":"2024-03-18T14:41:40.559616+0000","last_peered":"2024-03-18T14:41:40.559616+0000","last_clean":"2024-03-18T14:41:40.559616+0000","last_became_active":"2024-03-18T14:41:35.550839+0000","last_became_peered":"2024-03-18T14:41:35.550839+0000","last_unstale":"2024-03-18T14:41:40.559616+0000","last_undegraded":"2024-03-18T14:41:40.559616+0000","last_fullsized":"2024-03-18T14:41:40.559616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779023+0000","last_change":"2024-03-18T14:41:35.554787+0000","last_active":"2024-03-18T14:41:40.779023+0000","last_peered":"2024-03-18T14:41:40.779023+0000","last_clean":"2024-03-18T14:41:40.779023+0000","last_became_active":"2024-03-18T14:41:35.554484+0000","last_became_peered":"2024-03-18T14:41:35.554484+0000","last_unstale":"2024-03-18T14:41:40.779023+0000","last_undegraded":"2024-03-18T14:41:40.779023+0000","last_fullsized":"2024-03-18T14:41:40.779023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564088+0000","last_change":"2024-03-18T14:41:35.544958+0000","last_active":"2024-03-18T14:41:40.564088+0000","last_peered":"2024-03-18T14:41:40.564088+0000","last_clean":"2024-03-18T14:41:40.564088+0000","last_became_active":"2024-03-18T14:41:35.544600+0000","last_became_peered":"2024-03-18T14:41:35.544600+0000","last_unstale":"2024-03-18T14:41:40.564088+0000","last_undegraded":"2024-03-18T14:41:40.564088+0000","last_fullsized":"2024-03-18T14:41:40.564088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778046+0000","last_change":"2024-03-18T14:41:35.549309+0000","last_active":"2024-03-18T14:41:40.778046+0000","last_peered":"2024-03-18T14:41:40.778046+0000","last_clean":"2024-03-18T14:41:40.778046+0000","last_became_active":"2024-03-18T14:41:35.549039+0000","last_became_peered":"2024-03-18T14:41:35.549039+0000","last_unstale":"2024-03-18T14:41:40.778046+0000","last_undegraded":"2024-03-18T14:41:40.778046+0000","last_fullsized":"2024-03-18T14:41:40.778046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778908+0000","last_change":"2024-03-18T14:41:35.563387+0000","last_active":"2024-03-18T14:41:40.778908+0000","last_peered":"2024-03-18T14:41:40.778908+0000","last_clean":"2024-03-18T14:41:40.778908+0000","last_became_active":"2024-03-18T14:41:35.562621+0000","last_became_peered":"2024-03-18T14:41:35.562621+0000","last_unstale":"2024-03-18T14:41:40.778908+0000","last_undegraded":"2024-03-18T14:41:40.778908+0000","last_fullsized":"2024-03-18T14:41:40.778908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564514+0000","last_change":"2024-03-18T14:41:35.549988+0000","last_active":"2024-03-18T14:41:40.564514+0000","last_peered":"2024-03-18T14:41:40.564514+0000","last_clean":"2024-03-18T14:41:40.564514+0000","last_became_active":"2024-03-18T14:41:35.549713+0000","last_became_peered":"2024-03-18T14:41:35.549713+0000","last_unstale":"2024-03-18T14:41:40.564514+0000","last_undegraded":"2024-03-18T14:41:40.564514+0000","last_fullsized":"2024-03-18T14:41:40.564514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559089+0000","last_change":"2024-03-18T14:41:35.540097+0000","last_active":"2024-03-18T14:41:40.559089+0000","last_peered":"2024-03-18T14:41:40.559089+0000","last_clean":"2024-03-18T14:41:40.559089+0000","last_became_active":"2024-03-18T14:41:35.539823+0000","last_became_peered":"2024-03-18T14:41:35.539823+0000","last_unstale":"2024-03-18T14:41:40.559089+0000","last_undegraded":"2024-03-18T14:41:40.559089+0000","last_fullsized":"2024-03-18T14:41:40.559089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563495+0000","last_change":"2024-03-18T14:41:35.544547+0000","last_active":"2024-03-18T14:41:40.563495+0000","last_peered":"2024-03-18T14:41:40.563495+0000","last_clean":"2024-03-18T14:41:40.563495+0000","last_became_active":"2024-03-18T14:41:35.544026+0000","last_became_peered":"2024-03-18T14:41:35.544026+0000","last_unstale":"2024-03-18T14:41:40.563495+0000","last_undegraded":"2024-03-18T14:41:40.563495+0000","last_fullsized":"2024-03-18T14:41:40.563495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559538+0000","last_change":"2024-03-18T14:41:35.545773+0000","last_active":"2024-03-18T14:41:40.559538+0000","last_peered":"2024-03-18T14:41:40.559538+0000","last_clean":"2024-03-18T14:41:40.559538+0000","last_became_active":"2024-03-18T14:41:35.545449+0000","last_became_peered":"2024-03-18T14:41:35.545449+0000","last_unstale":"2024-03-18T14:41:40.559538+0000","last_undegraded":"2024-03-18T14:41:40.559538+0000","last_fullsized":"2024-03-18T14:41:40.559538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778786+0000","last_change":"2024-03-18T14:41:35.563339+0000","last_active":"2024-03-18T14:41:40.778786+0000","last_peered":"2024-03-18T14:41:40.778786+0000","last_clean":"2024-03-18T14:41:40.778786+0000","last_became_active":"2024-03-18T14:41:35.563117+0000","last_became_peered":"2024-03-18T14:41:35.563117+0000","last_unstale":"2024-03-18T14:41:40.778786+0000","last_undegraded":"2024-03-18T14:41:40.778786+0000","last_fullsized":"2024-03-18T14:41:40.778786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778522+0000","last_change":"2024-03-18T14:41:35.559014+0000","last_active":"2024-03-18T14:41:40.778522+0000","last_peered":"2024-03-18T14:41:40.778522+0000","last_clean":"2024-03-18T14:41:40.778522+0000","last_became_active":"2024-03-18T14:41:35.558349+0000","last_became_peered":"2024-03-18T14:41:35.558349+0000","last_unstale":"2024-03-18T14:41:40.778522+0000","last_undegraded":"2024-03-18T14:41:40.778522+0000","last_fullsized":"2024-03-18T14:41:40.778522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778971+0000","last_change":"2024-03-18T14:41:35.545841+0000","last_active":"2024-03-18T14:41:40.778971+0000","last_peered":"2024-03-18T14:41:40.778971+0000","last_clean":"2024-03-18T14:41:40.778971+0000","last_became_active":"2024-03-18T14:41:35.545543+0000","last_became_peered":"2024-03-18T14:41:35.545543+0000","last_unstale":"2024-03-18T14:41:40.778971+0000","last_undegraded":"2024-03-18T14:41:40.778971+0000","last_fullsized":"2024-03-18T14:41:40.778971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778782+0000","last_change":"2024-03-18T14:41:35.551018+0000","last_active":"2024-03-18T14:41:40.778782+0000","last_peered":"2024-03-18T14:41:40.778782+0000","last_clean":"2024-03-18T14:41:40.778782+0000","last_became_active":"2024-03-18T14:41:35.550737+0000","last_became_peered":"2024-03-18T14:41:35.550737+0000","last_unstale":"2024-03-18T14:41:40.778782+0000","last_undegraded":"2024-03-18T14:41:40.778782+0000","last_fullsized":"2024-03-18T14:41:40.778782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560481+0000","last_change":"2024-03-18T14:41:35.547133+0000","last_active":"2024-03-18T14:41:40.560481+0000","last_peered":"2024-03-18T14:41:40.560481+0000","last_clean":"2024-03-18T14:41:40.560481+0000","last_became_active":"2024-03-18T14:41:35.546778+0000","last_became_peered":"2024-03-18T14:41:35.546778+0000","last_unstale":"2024-03-18T14:41:40.560481+0000","last_undegraded":"2024-03-18T14:41:40.560481+0000","last_fullsized":"2024-03-18T14:41:40.560481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559067+0000","last_change":"2024-03-18T14:41:35.550366+0000","last_active":"2024-03-18T14:41:40.559067+0000","last_peered":"2024-03-18T14:41:40.559067+0000","last_clean":"2024-03-18T14:41:40.559067+0000","last_became_active":"2024-03-18T14:41:35.550111+0000","last_became_peered":"2024-03-18T14:41:35.550111+0000","last_unstale":"2024-03-18T14:41:40.559067+0000","last_undegraded":"2024-03-18T14:41:40.559067+0000","last_fullsized":"2024-03-18T14:41:40.559067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778058+0000","last_change":"2024-03-18T14:41:35.536085+0000","last_active":"2024-03-18T14:41:40.778058+0000","last_peered":"2024-03-18T14:41:40.778058+0000","last_clean":"2024-03-18T14:41:40.778058+0000","last_became_active":"2024-03-18T14:41:35.535769+0000","last_became_peered":"2024-03-18T14:41:35.535769+0000","last_unstale":"2024-03-18T14:41:40.778058+0000","last_undegraded":"2024-03-18T14:41:40.778058+0000","last_fullsized":"2024-03-18T14:41:40.778058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778571+0000","last_change":"2024-03-18T14:41:35.550291+0000","last_active":"2024-03-18T14:41:40.778571+0000","last_peered":"2024-03-18T14:41:40.778571+0000","last_clean":"2024-03-18T14:41:40.778571+0000","last_became_active":"2024-03-18T14:41:35.549994+0000","last_became_peered":"2024-03-18T14:41:35.549994+0000","last_unstale":"2024-03-18T14:41:40.778571+0000","last_undegraded":"2024-03-18T14:41:40.778571+0000","last_fullsized":"2024-03-18T14:41:40.778571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:20.446494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563821+0000","last_change":"2024-03-18T14:41:35.551464+0000","last_active":"2024-03-18T14:41:40.563821+0000","last_peered":"2024-03-18T14:41:40.563821+0000","last_clean":"2024-03-18T14:41:40.563821+0000","last_became_active":"2024-03-18T14:41:35.551200+0000","last_became_peered":"2024-03-18T14:41:35.551200+0000","last_unstale":"2024-03-18T14:41:40.563821+0000","last_undegraded":"2024-03-18T14:41:40.563821+0000","last_fullsized":"2024-03-18T14:41:40.563821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559869+0000","last_change":"2024-03-18T14:41:35.558583+0000","last_active":"2024-03-18T14:41:40.559869+0000","last_peered":"2024-03-18T14:41:40.559869+0000","last_clean":"2024-03-18T14:41:40.559869+0000","last_became_active":"2024-03-18T14:41:35.557388+0000","last_became_peered":"2024-03-18T14:41:35.557388+0000","last_unstale":"2024-03-18T14:41:40.559869+0000","last_undegraded":"2024-03-18T14:41:40.559869+0000","last_fullsized":"2024-03-18T14:41:40.559869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778685+0000","last_change":"2024-03-18T14:41:35.556362+0000","last_active":"2024-03-18T14:41:40.778685+0000","last_peered":"2024-03-18T14:41:40.778685+0000","last_clean":"2024-03-18T14:41:40.778685+0000","last_became_active":"2024-03-18T14:41:35.556073+0000","last_became_peered":"2024-03-18T14:41:35.556073+0000","last_unstale":"2024-03-18T14:41:40.778685+0000","last_undegraded":"2024-03-18T14:41:40.778685+0000","last_fullsized":"2024-03-18T14:41:40.778685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778827+0000","last_change":"2024-03-18T14:41:35.553776+0000","last_active":"2024-03-18T14:41:40.778827+0000","last_peered":"2024-03-18T14:41:40.778827+0000","last_clean":"2024-03-18T14:41:40.778827+0000","last_became_active":"2024-03-18T14:41:35.553413+0000","last_became_peered":"2024-03-18T14:41:35.553413+0000","last_unstale":"2024-03-18T14:41:40.778827+0000","last_undegraded":"2024-03-18T14:41:40.778827+0000","last_fullsized":"2024-03-18T14:41:40.778827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778851+0000","last_change":"2024-03-18T14:41:35.556025+0000","last_active":"2024-03-18T14:41:40.778851+0000","last_peered":"2024-03-18T14:41:40.778851+0000","last_clean":"2024-03-18T14:41:40.778851+0000","last_became_active":"2024-03-18T14:41:35.555322+0000","last_became_peered":"2024-03-18T14:41:35.555322+0000","last_unstale":"2024-03-18T14:41:40.778851+0000","last_undegraded":"2024-03-18T14:41:40.778851+0000","last_fullsized":"2024-03-18T14:41:40.778851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563141+0000","last_change":"2024-03-18T14:41:35.545290+0000","last_active":"2024-03-18T14:41:40.563141+0000","last_peered":"2024-03-18T14:41:40.563141+0000","last_clean":"2024-03-18T14:41:40.563141+0000","last_became_active":"2024-03-18T14:41:35.543962+0000","last_became_peered":"2024-03-18T14:41:35.543962+0000","last_unstale":"2024-03-18T14:41:40.563141+0000","last_undegraded":"2024-03-18T14:41:40.563141+0000","last_fullsized":"2024-03-18T14:41:40.563141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778288+0000","last_change":"2024-03-18T14:41:35.556987+0000","last_active":"2024-03-18T14:41:40.778288+0000","last_peered":"2024-03-18T14:41:40.778288+0000","last_clean":"2024-03-18T14:41:40.778288+0000","last_became_active":"2024-03-18T14:41:35.556713+0000","last_became_peered":"2024-03-18T14:41:35.556713+0000","last_unstale":"2024-03-18T14:41:40.778288+0000","last_undegraded":"2024-03-18T14:41:40.778288+0000","last_fullsized":"2024-03-18T14:41:40.778288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559461+0000","last_change":"2024-03-18T14:41:35.548724+0000","last_active":"2024-03-18T14:41:40.559461+0000","last_peered":"2024-03-18T14:41:40.559461+0000","last_clean":"2024-03-18T14:41:40.559461+0000","last_became_active":"2024-03-18T14:41:35.547693+0000","last_became_peered":"2024-03-18T14:41:35.547693+0000","last_unstale":"2024-03-18T14:41:40.559461+0000","last_undegraded":"2024-03-18T14:41:40.559461+0000","last_fullsized":"2024-03-18T14:41:40.559461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:35.544193+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:35.543909+0000","last_became_peered":"2024-03-18T14:41:35.543909+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559445+0000","last_change":"2024-03-18T14:41:35.549929+0000","last_active":"2024-03-18T14:41:40.559445+0000","last_peered":"2024-03-18T14:41:40.559445+0000","last_clean":"2024-03-18T14:41:40.559445+0000","last_became_active":"2024-03-18T14:41:35.547332+0000","last_became_peered":"2024-03-18T14:41:35.547332+0000","last_unstale":"2024-03-18T14:41:40.559445+0000","last_undegraded":"2024-03-18T14:41:40.559445+0000","last_fullsized":"2024-03-18T14:41:40.559445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779000+0000","last_change":"2024-03-18T14:41:35.550445+0000","last_active":"2024-03-18T14:41:40.779000+0000","last_peered":"2024-03-18T14:41:40.779000+0000","last_clean":"2024-03-18T14:41:40.779000+0000","last_became_active":"2024-03-18T14:41:35.550120+0000","last_became_peered":"2024-03-18T14:41:35.550120+0000","last_unstale":"2024-03-18T14:41:40.779000+0000","last_undegraded":"2024-03-18T14:41:40.779000+0000","last_fullsized":"2024-03-18T14:41:40.779000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778964+0000","last_change":"2024-03-18T14:41:35.548653+0000","last_active":"2024-03-18T14:41:40.778964+0000","last_peered":"2024-03-18T14:41:40.778964+0000","last_clean":"2024-03-18T14:41:40.778964+0000","last_became_active":"2024-03-18T14:41:35.548305+0000","last_became_peered":"2024-03-18T14:41:35.548305+0000","last_unstale":"2024-03-18T14:41:40.778964+0000","last_undegraded":"2024-03-18T14:41:40.778964+0000","last_fullsized":"2024-03-18T14:41:40.778964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778662+0000","last_change":"2024-03-18T14:41:35.560778+0000","last_active":"2024-03-18T14:41:40.778662+0000","last_peered":"2024-03-18T14:41:40.778662+0000","last_clean":"2024-03-18T14:41:40.778662+0000","last_became_active":"2024-03-18T14:41:35.560478+0000","last_became_peered":"2024-03-18T14:41:35.560478+0000","last_unstale":"2024-03-18T14:41:40.778662+0000","last_undegraded":"2024-03-18T14:41:40.778662+0000","last_fullsized":"2024-03-18T14:41:40.778662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:41:58.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:41:58.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.743+0000 7f7d45636640 1 Processor -- start 2024-03-18T14:41:58.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.743+0000 7f7d45636640 1 -- start start 2024-03-18T14:41:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d45636640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7d4015e020 0x7f7d40162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d45636640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d400ced00 0x7f7d400d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d45636640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d400b9660 con 0x7f7d400d3210 2024-03-18T14:41:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d45636640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7d400ce390 con 0x7f7d400ced00 2024-03-18T14:41:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d45636640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7d400b9900 con 0x7f7d4015e020 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3e7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d400ced00 0x7f7d400d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7d400d3210 0x7f7d400d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53032/0 (socket says 172.21.15.6:53032) 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3e7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d400ced00 0x7f7d400d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56326/0 (socket says 172.21.15.6:56326) 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3f7fe640 1 -- 172.21.15.6:0/3089383670 learned_addr learned my addr 172.21.15.6:0/3089383670 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3effd640 1 --2- 172.21.15.6:0/3089383670 >> v2:172.21.15.92:3300/0 conn(0x7f7d4015e020 0x7f7d40162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3e7fc640 1 -- 172.21.15.6:0/3089383670 >> v2:172.21.15.92:3300/0 conn(0x7f7d4015e020 msgr2=0x7f7d40162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3e7fc640 1 --2- 172.21.15.6:0/3089383670 >> v2:172.21.15.92:3300/0 conn(0x7f7d4015e020 0x7f7d40162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3e7fc640 1 -- 172.21.15.6:0/3089383670 >> v1:172.21.15.6:6789/0 conn(0x7f7d400d3210 legacy=0x7f7d400d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3e7fc640 1 -- 172.21.15.6:0/3089383670 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d400b9be0 con 0x7f7d400ced00 2024-03-18T14:41:58.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.744+0000 7f7d3dffb640 1 -- 172.21.15.6:0/3089383670 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224081770 0 0) 0x7f7d400b9660 con 0x7f7d400d3210 2024-03-18T14:41:58.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.745+0000 7f7d3e7fc640 1 --2- 172.21.15.6:0/3089383670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d400d1b60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d400d1450 tx=0x7f7d2802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a88d24aef2af09d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.745+0000 7f7d3dffb640 1 -- 172.21.15.6:0/3089383670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d280315a0 con 0x7f7d400ced00 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.745+0000 7f7d3dffb640 1 -- 172.21.15.6:0/3089383670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d28037070 con 0x7f7d400ced00 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.745+0000 7f7d3dffb640 1 -- 172.21.15.6:0/3089383670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28031960 con 0x7f7d400ced00 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 -- 172.21.15.6:0/3089383670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 msgr2=0x7f7d400d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 --2- 172.21.15.6:0/3089383670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d400d1b60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d400d1450 tx=0x7f7d2802b840 comp rx=0 tx=0).stop 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 -- 172.21.15.6:0/3089383670 shutdown_connections 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 --2- 172.21.15.6:0/3089383670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d400d1b60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 --2- 172.21.15.6:0/3089383670 >> v2:172.21.15.92:3300/0 conn(0x7f7d4015e020 0x7f7d40162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 -- 172.21.15.6:0/3089383670 >> 172.21.15.6:0/3089383670 conn(0x7f7d40072ce0 msgr2=0x7f7d400730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 -- 172.21.15.6:0/3089383670 shutdown_connections 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.746+0000 7f7d45636640 1 -- 172.21.15.6:0/3089383670 wait complete. 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.747+0000 7f7d45636640 1 Processor -- start 2024-03-18T14:41:58.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.747+0000 7f7d45636640 1 -- start start 2024-03-18T14:41:58.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d45636640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d4015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d45636640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d400d3210 0x7f7d4015c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d45636640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d4015e020 0x7f7d401f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d45636640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7d400cf440 con 0x7f7d400ced00 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d45636640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7d400cf2c0 con 0x7f7d400d3210 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d45636640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7d400d20a0 con 0x7f7d4015e020 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d4015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d4015ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56330/0 (socket says 172.21.15.6:56330) 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 -- 172.21.15.6:0/2812305137 learned_addr learned my addr 172.21.15.6:0/2812305137 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3e7fc640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d400d3210 0x7f7d4015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3f7fe640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d4015e020 0x7f7d401f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 -- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d4015e020 msgr2=0x7f7d401f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d4015e020 0x7f7d401f3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 -- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d400d3210 msgr2=0x7f7d4015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d400d3210 0x7f7d4015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3effd640 1 -- 172.21.15.6:0/2812305137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4015f380 con 0x7f7d400ced00 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.748+0000 7f7d3e7fc640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d400d3210 0x7f7d4015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.749+0000 7f7d3effd640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d4015ee40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000efc0 tx=0x7f7d3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.749+0000 7f7d377fe640 1 -- 172.21.15.6:0/2812305137 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d30018020 con 0x7f7d400ced00 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.749+0000 7f7d377fe640 1 -- 172.21.15.6:0/2812305137 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d30014480 con 0x7f7d400ced00 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.749+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d400b3a90 con 0x7f7d400ced00 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.749+0000 7f7d377fe640 1 -- 172.21.15.6:0/2812305137 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d30014850 con 0x7f7d400ced00 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.749+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d400b4330 con 0x7f7d400ced00 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.750+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d400b3e80 con 0x7f7d400ced00 2024-03-18T14:41:58.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.750+0000 7f7d377fe640 1 -- 172.21.15.6:0/2812305137 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7d30005d10 con 0x7f7d400ced00 2024-03-18T14:41:58.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.751+0000 7f7d377fe640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:58.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.751+0000 7f7d377fe640 1 -- 172.21.15.6:0/2812305137 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7d3008b950 con 0x7f7d400ced00 2024-03-18T14:41:58.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.751+0000 7f7d3e7fc640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:58.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.751+0000 7f7d3e7fc640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7d28036000 tx=0x7f7d28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:58.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.755+0000 7f7d377fe640 1 -- 172.21.15.6:0/2812305137 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7d30022030 con 0x7f7d400ced00 2024-03-18T14:41:58.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.933+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d4004f620 con 0x7f7d2006bbd0 2024-03-18T14:41:58.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.941+0000 7f7d377fe640 1 -- 172.21.15.6:0/2812305137 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f7d4004f620 con 0x7f7d2006bbd0 2024-03-18T14:41:58.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:41:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 msgr2=0x7f7d2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7d28036000 tx=0x7f7d28008030 comp rx=0 tx=0).stop 2024-03-18T14:41:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 msgr2=0x7f7d4015ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d4015ee40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000efc0 tx=0x7f7d3000c490 comp rx=0 tx=0).stop 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 shutdown_connections 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d4015e020 0x7f7d401f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d400d3210 0x7f7d4015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 --2- 172.21.15.6:0/2812305137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d400ced00 0x7f7d4015ee40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 >> 172.21.15.6:0/2812305137 conn(0x7f7d40072ce0 msgr2=0x7f7d400ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 shutdown_connections 2024-03-18T14:41:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:58.944+0000 7f7d45636640 1 -- 172.21.15.6:0/2812305137 wait complete. 2024-03-18T14:41:58.957 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-03-18T14:41:57.384459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002613"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779658+0000","last_change":"2024-03-18T14:41:34.533975+0000","last_active":"2024-03-18T14:41:40.779658+0000","last_peered":"2024-03-18T14:41:40.779658+0000","last_clean":"2024-03-18T14:41:40.779658+0000","last_became_active":"2024-03-18T14:41:34.533697+0000","last_became_peered":"2024-03-18T14:41:34.533697+0000","last_unstale":"2024-03-18T14:41:40.779658+0000","last_undegraded":"2024-03-18T14:41:40.779658+0000","last_fullsized":"2024-03-18T14:41:40.779658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:35.555471+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:35.555193+0000","last_became_peered":"2024-03-18T14:41:35.555193+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563408+0000","last_change":"2024-03-18T14:41:39.604430+0000","last_active":"2024-03-18T14:41:40.563408+0000","last_peered":"2024-03-18T14:41:40.563408+0000","last_clean":"2024-03-18T14:41:40.563408+0000","last_became_active":"2024-03-18T14:41:39.603958+0000","last_became_peered":"2024-03-18T14:41:39.603958+0000","last_unstale":"2024-03-18T14:41:40.563408+0000","last_undegraded":"2024-03-18T14:41:40.563408+0000","last_fullsized":"2024-03-18T14:41:40.563408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:17.770515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564369+0000","last_change":"2024-03-18T14:41:34.535172+0000","last_active":"2024-03-18T14:41:40.564369+0000","last_peered":"2024-03-18T14:41:40.564369+0000","last_clean":"2024-03-18T14:41:40.564369+0000","last_became_active":"2024-03-18T14:41:34.534849+0000","last_became_peered":"2024-03-18T14:41:34.534849+0000","last_unstale":"2024-03-18T14:41:40.564369+0000","last_undegraded":"2024-03-18T14:41:40.564369+0000","last_fullsized":"2024-03-18T14:41:40.564369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779347+0000","last_change":"2024-03-18T14:41:35.553407+0000","last_active":"2024-03-18T14:41:40.779347+0000","last_peered":"2024-03-18T14:41:40.779347+0000","last_clean":"2024-03-18T14:41:40.779347+0000","last_became_active":"2024-03-18T14:41:35.553091+0000","last_became_peered":"2024-03-18T14:41:35.553091+0000","last_unstale":"2024-03-18T14:41:40.779347+0000","last_undegraded":"2024-03-18T14:41:40.779347+0000","last_fullsized":"2024-03-18T14:41:40.779347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559102+0000","last_change":"2024-03-18T14:41:39.579025+0000","last_active":"2024-03-18T14:41:40.559102+0000","last_peered":"2024-03-18T14:41:40.559102+0000","last_clean":"2024-03-18T14:41:40.559102+0000","last_became_active":"2024-03-18T14:41:39.578650+0000","last_became_peered":"2024-03-18T14:41:39.578650+0000","last_unstale":"2024-03-18T14:41:40.559102+0000","last_undegraded":"2024-03-18T14:41:40.559102+0000","last_fullsized":"2024-03-18T14:41:40.559102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:53.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778929+0000","last_change":"2024-03-18T14:41:34.536627+0000","last_active":"2024-03-18T14:41:40.778929+0000","last_peered":"2024-03-18T14:41:40.778929+0000","last_clean":"2024-03-18T14:41:40.778929+0000","last_became_active":"2024-03-18T14:41:34.536136+0000","last_became_peered":"2024-03-18T14:41:34.536136+0000","last_unstale":"2024-03-18T14:41:40.778929+0000","last_undegraded":"2024-03-18T14:41:40.778929+0000","last_fullsized":"2024-03-18T14:41:40.778929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:12.036369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559536+0000","last_change":"2024-03-18T14:41:35.544918+0000","last_active":"2024-03-18T14:41:40.559536+0000","last_peered":"2024-03-18T14:41:40.559536+0000","last_clean":"2024-03-18T14:41:40.559536+0000","last_became_active":"2024-03-18T14:41:35.544612+0000","last_became_peered":"2024-03-18T14:41:35.544612+0000","last_unstale":"2024-03-18T14:41:40.559536+0000","last_undegraded":"2024-03-18T14:41:40.559536+0000","last_fullsized":"2024-03-18T14:41:40.559536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560251+0000","last_change":"2024-03-18T14:41:39.580665+0000","last_active":"2024-03-18T14:41:40.560251+0000","last_peered":"2024-03-18T14:41:40.560251+0000","last_clean":"2024-03-18T14:41:40.560251+0000","last_became_active":"2024-03-18T14:41:39.580231+0000","last_became_peered":"2024-03-18T14:41:39.580231+0000","last_unstale":"2024-03-18T14:41:40.560251+0000","last_undegraded":"2024-03-18T14:41:40.560251+0000","last_fullsized":"2024-03-18T14:41:40.560251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778491+0000","last_change":"2024-03-18T14:41:34.531008+0000","last_active":"2024-03-18T14:41:40.778491+0000","last_peered":"2024-03-18T14:41:40.778491+0000","last_clean":"2024-03-18T14:41:40.778491+0000","last_became_active":"2024-03-18T14:41:34.530587+0000","last_became_peered":"2024-03-18T14:41:34.530587+0000","last_unstale":"2024-03-18T14:41:40.778491+0000","last_undegraded":"2024-03-18T14:41:40.778491+0000","last_fullsized":"2024-03-18T14:41:40.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560259+0000","last_change":"2024-03-18T14:41:35.542756+0000","last_active":"2024-03-18T14:41:40.560259+0000","last_peered":"2024-03-18T14:41:40.560259+0000","last_clean":"2024-03-18T14:41:40.560259+0000","last_became_active":"2024-03-18T14:41:35.542486+0000","last_became_peered":"2024-03-18T14:41:35.542486+0000","last_unstale":"2024-03-18T14:41:40.560259+0000","last_undegraded":"2024-03-18T14:41:40.560259+0000","last_fullsized":"2024-03-18T14:41:40.560259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559437+0000","last_change":"2024-03-18T14:41:39.578229+0000","last_active":"2024-03-18T14:41:40.559437+0000","last_peered":"2024-03-18T14:41:40.559437+0000","last_clean":"2024-03-18T14:41:40.559437+0000","last_became_active":"2024-03-18T14:41:39.577812+0000","last_became_peered":"2024-03-18T14:41:39.577812+0000","last_unstale":"2024-03-18T14:41:40.559437+0000","last_undegraded":"2024-03-18T14:41:40.559437+0000","last_fullsized":"2024-03-18T14:41:40.559437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:50.123586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778041+0000","last_change":"2024-03-18T14:41:35.549686+0000","last_active":"2024-03-18T14:41:40.778041+0000","last_peered":"2024-03-18T14:41:40.778041+0000","last_clean":"2024-03-18T14:41:40.778041+0000","last_became_active":"2024-03-18T14:41:35.549240+0000","last_became_peered":"2024-03-18T14:41:35.549240+0000","last_unstale":"2024-03-18T14:41:40.778041+0000","last_undegraded":"2024-03-18T14:41:40.778041+0000","last_fullsized":"2024-03-18T14:41:40.778041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778484+0000","last_change":"2024-03-18T14:41:34.530798+0000","last_active":"2024-03-18T14:41:40.778484+0000","last_peered":"2024-03-18T14:41:40.778484+0000","last_clean":"2024-03-18T14:41:40.778484+0000","last_became_active":"2024-03-18T14:41:34.530415+0000","last_became_peered":"2024-03-18T14:41:34.530415+0000","last_unstale":"2024-03-18T14:41:40.778484+0000","last_undegraded":"2024-03-18T14:41:40.778484+0000","last_fullsized":"2024-03-18T14:41:40.778484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560204+0000","last_change":"2024-03-18T14:41:39.584982+0000","last_active":"2024-03-18T14:41:40.560204+0000","last_peered":"2024-03-18T14:41:40.560204+0000","last_clean":"2024-03-18T14:41:40.560204+0000","last_became_active":"2024-03-18T14:41:39.584630+0000","last_became_peered":"2024-03-18T14:41:39.584630+0000","last_unstale":"2024-03-18T14:41:40.560204+0000","last_undegraded":"2024-03-18T14:41:40.560204+0000","last_fullsized":"2024-03-18T14:41:40.560204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563830+0000","last_change":"2024-03-18T14:41:35.552305+0000","last_active":"2024-03-18T14:41:40.563830+0000","last_peered":"2024-03-18T14:41:40.563830+0000","last_clean":"2024-03-18T14:41:40.563830+0000","last_became_active":"2024-03-18T14:41:35.552031+0000","last_became_peered":"2024-03-18T14:41:35.552031+0000","last_unstale":"2024-03-18T14:41:40.563830+0000","last_undegraded":"2024-03-18T14:41:40.563830+0000","last_fullsized":"2024-03-18T14:41:40.563830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779939+0000","last_change":"2024-03-18T14:41:34.530930+0000","last_active":"2024-03-18T14:41:40.779939+0000","last_peered":"2024-03-18T14:41:40.779939+0000","last_clean":"2024-03-18T14:41:40.779939+0000","last_became_active":"2024-03-18T14:41:34.530645+0000","last_became_peered":"2024-03-18T14:41:34.530645+0000","last_unstale":"2024-03-18T14:41:40.779939+0000","last_undegraded":"2024-03-18T14:41:40.779939+0000","last_fullsized":"2024-03-18T14:41:40.779939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559457+0000","last_change":"2024-03-18T14:41:39.585926+0000","last_active":"2024-03-18T14:41:40.559457+0000","last_peered":"2024-03-18T14:41:40.559457+0000","last_clean":"2024-03-18T14:41:40.559457+0000","last_became_active":"2024-03-18T14:41:39.585465+0000","last_became_peered":"2024-03-18T14:41:39.585465+0000","last_unstale":"2024-03-18T14:41:40.559457+0000","last_undegraded":"2024-03-18T14:41:40.559457+0000","last_fullsized":"2024-03-18T14:41:40.559457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563386+0000","last_change":"2024-03-18T14:41:35.541836+0000","last_active":"2024-03-18T14:41:40.563386+0000","last_peered":"2024-03-18T14:41:40.563386+0000","last_clean":"2024-03-18T14:41:40.563386+0000","last_became_active":"2024-03-18T14:41:35.541532+0000","last_became_peered":"2024-03-18T14:41:35.541532+0000","last_unstale":"2024-03-18T14:41:40.563386+0000","last_undegraded":"2024-03-18T14:41:40.563386+0000","last_fullsized":"2024-03-18T14:41:40.563386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778143+0000","last_change":"2024-03-18T14:41:34.529169+0000","last_active":"2024-03-18T14:41:40.778143+0000","last_peered":"2024-03-18T14:41:40.778143+0000","last_clean":"2024-03-18T14:41:40.778143+0000","last_became_active":"2024-03-18T14:41:34.526870+0000","last_became_peered":"2024-03-18T14:41:34.526870+0000","last_unstale":"2024-03-18T14:41:40.778143+0000","last_undegraded":"2024-03-18T14:41:40.778143+0000","last_fullsized":"2024-03-18T14:41:40.778143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779714+0000","last_change":"2024-03-18T14:41:39.603928+0000","last_active":"2024-03-18T14:41:40.779714+0000","last_peered":"2024-03-18T14:41:40.779714+0000","last_clean":"2024-03-18T14:41:40.779714+0000","last_became_active":"2024-03-18T14:41:39.603236+0000","last_became_peered":"2024-03-18T14:41:39.603236+0000","last_unstale":"2024-03-18T14:41:40.779714+0000","last_undegraded":"2024-03-18T14:41:40.779714+0000","last_fullsized":"2024-03-18T14:41:40.779714+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559885+0000","last_change":"2024-03-18T14:41:35.538978+0000","last_active":"2024-03-18T14:41:40.559885+0000","last_peered":"2024-03-18T14:41:40.559885+0000","last_clean":"2024-03-18T14:41:40.559885+0000","last_became_active":"2024-03-18T14:41:35.538570+0000","last_became_peered":"2024-03-18T14:41:35.538570+0000","last_unstale":"2024-03-18T14:41:40.559885+0000","last_undegraded":"2024-03-18T14:41:40.559885+0000","last_fullsized":"2024-03-18T14:41:40.559885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:38.574781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777921+0000","last_change":"2024-03-18T14:41:34.537652+0000","last_active":"2024-03-18T14:41:40.777921+0000","last_peered":"2024-03-18T14:41:40.777921+0000","last_clean":"2024-03-18T14:41:40.777921+0000","last_became_active":"2024-03-18T14:41:34.537411+0000","last_became_peered":"2024-03-18T14:41:34.537411+0000","last_unstale":"2024-03-18T14:41:40.777921+0000","last_undegraded":"2024-03-18T14:41:40.777921+0000","last_fullsized":"2024-03-18T14:41:40.777921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560190+0000","last_change":"2024-03-18T14:41:39.580821+0000","last_active":"2024-03-18T14:41:40.560190+0000","last_peered":"2024-03-18T14:41:40.560190+0000","last_clean":"2024-03-18T14:41:40.560190+0000","last_became_active":"2024-03-18T14:41:39.580317+0000","last_became_peered":"2024-03-18T14:41:39.580317+0000","last_unstale":"2024-03-18T14:41:40.560190+0000","last_undegraded":"2024-03-18T14:41:40.560190+0000","last_fullsized":"2024-03-18T14:41:40.560190+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778799+0000","last_change":"2024-03-18T14:41:35.552139+0000","last_active":"2024-03-18T14:41:40.778799+0000","last_peered":"2024-03-18T14:41:40.778799+0000","last_clean":"2024-03-18T14:41:40.778799+0000","last_became_active":"2024-03-18T14:41:35.551809+0000","last_became_peered":"2024-03-18T14:41:35.551809+0000","last_unstale":"2024-03-18T14:41:40.778799+0000","last_undegraded":"2024-03-18T14:41:40.778799+0000","last_fullsized":"2024-03-18T14:41:40.778799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779124+0000","last_change":"2024-03-18T14:41:34.534282+0000","last_active":"2024-03-18T14:41:40.779124+0000","last_peered":"2024-03-18T14:41:40.779124+0000","last_clean":"2024-03-18T14:41:40.779124+0000","last_became_active":"2024-03-18T14:41:34.533937+0000","last_became_peered":"2024-03-18T14:41:34.533937+0000","last_unstale":"2024-03-18T14:41:40.779124+0000","last_undegraded":"2024-03-18T14:41:40.779124+0000","last_fullsized":"2024-03-18T14:41:40.779124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559601+0000","last_change":"2024-03-18T14:41:39.602565+0000","last_active":"2024-03-18T14:41:40.559601+0000","last_peered":"2024-03-18T14:41:40.559601+0000","last_clean":"2024-03-18T14:41:40.559601+0000","last_became_active":"2024-03-18T14:41:39.602164+0000","last_became_peered":"2024-03-18T14:41:39.602164+0000","last_unstale":"2024-03-18T14:41:40.559601+0000","last_undegraded":"2024-03-18T14:41:40.559601+0000","last_fullsized":"2024-03-18T14:41:40.559601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564632+0000","last_change":"2024-03-18T14:41:35.547600+0000","last_active":"2024-03-18T14:41:40.564632+0000","last_peered":"2024-03-18T14:41:40.564632+0000","last_clean":"2024-03-18T14:41:40.564632+0000","last_became_active":"2024-03-18T14:41:35.547324+0000","last_became_peered":"2024-03-18T14:41:35.547324+0000","last_unstale":"2024-03-18T14:41:40.564632+0000","last_undegraded":"2024-03-18T14:41:40.564632+0000","last_fullsized":"2024-03-18T14:41:40.564632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779055+0000","last_change":"2024-03-18T14:41:34.537441+0000","last_active":"2024-03-18T14:41:40.779055+0000","last_peered":"2024-03-18T14:41:40.779055+0000","last_clean":"2024-03-18T14:41:40.779055+0000","last_became_active":"2024-03-18T14:41:34.536599+0000","last_became_peered":"2024-03-18T14:41:34.536599+0000","last_unstale":"2024-03-18T14:41:40.779055+0000","last_undegraded":"2024-03-18T14:41:40.779055+0000","last_fullsized":"2024-03-18T14:41:40.779055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563656+0000","last_change":"2024-03-18T14:41:39.604429+0000","last_active":"2024-03-18T14:41:40.563656+0000","last_peered":"2024-03-18T14:41:40.563656+0000","last_clean":"2024-03-18T14:41:40.563656+0000","last_became_active":"2024-03-18T14:41:39.604046+0000","last_became_peered":"2024-03-18T14:41:39.604046+0000","last_unstale":"2024-03-18T14:41:40.563656+0000","last_undegraded":"2024-03-18T14:41:40.563656+0000","last_fullsized":"2024-03-18T14:41:40.563656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:21.835327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778649+0000","last_change":"2024-03-18T14:41:35.539912+0000","last_active":"2024-03-18T14:41:40.778649+0000","last_peered":"2024-03-18T14:41:40.778649+0000","last_clean":"2024-03-18T14:41:40.778649+0000","last_became_active":"2024-03-18T14:41:35.538765+0000","last_became_peered":"2024-03-18T14:41:35.538765+0000","last_unstale":"2024-03-18T14:41:40.778649+0000","last_undegraded":"2024-03-18T14:41:40.778649+0000","last_fullsized":"2024-03-18T14:41:40.778649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779021+0000","last_change":"2024-03-18T14:41:34.537358+0000","last_active":"2024-03-18T14:41:40.779021+0000","last_peered":"2024-03-18T14:41:40.779021+0000","last_clean":"2024-03-18T14:41:40.779021+0000","last_became_active":"2024-03-18T14:41:34.537020+0000","last_became_peered":"2024-03-18T14:41:34.537020+0000","last_unstale":"2024-03-18T14:41:40.779021+0000","last_undegraded":"2024-03-18T14:41:40.779021+0000","last_fullsized":"2024-03-18T14:41:40.779021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:24.398074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778734+0000","last_change":"2024-03-18T14:41:39.601563+0000","last_active":"2024-03-18T14:41:40.778734+0000","last_peered":"2024-03-18T14:41:40.778734+0000","last_clean":"2024-03-18T14:41:40.778734+0000","last_became_active":"2024-03-18T14:41:39.600878+0000","last_became_peered":"2024-03-18T14:41:39.600878+0000","last_unstale":"2024-03-18T14:41:40.778734+0000","last_undegraded":"2024-03-18T14:41:40.778734+0000","last_fullsized":"2024-03-18T14:41:40.778734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:41.241691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560348+0000","last_change":"2024-03-18T14:41:35.551415+0000","last_active":"2024-03-18T14:41:40.560348+0000","last_peered":"2024-03-18T14:41:40.560348+0000","last_clean":"2024-03-18T14:41:40.560348+0000","last_became_active":"2024-03-18T14:41:35.550751+0000","last_became_peered":"2024-03-18T14:41:35.550751+0000","last_unstale":"2024-03-18T14:41:40.560348+0000","last_undegraded":"2024-03-18T14:41:40.560348+0000","last_fullsized":"2024-03-18T14:41:40.560348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778749+0000","last_change":"2024-03-18T14:41:34.534249+0000","last_active":"2024-03-18T14:41:40.778749+0000","last_peered":"2024-03-18T14:41:40.778749+0000","last_clean":"2024-03-18T14:41:40.778749+0000","last_became_active":"2024-03-18T14:41:34.533913+0000","last_became_peered":"2024-03-18T14:41:34.533913+0000","last_unstale":"2024-03-18T14:41:40.778749+0000","last_undegraded":"2024-03-18T14:41:40.778749+0000","last_fullsized":"2024-03-18T14:41:40.778749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:39.582572+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:39.581652+0000","last_became_peered":"2024-03-18T14:41:39.581652+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560327+0000","last_change":"2024-03-18T14:41:35.561297+0000","last_active":"2024-03-18T14:41:40.560327+0000","last_peered":"2024-03-18T14:41:40.560327+0000","last_clean":"2024-03-18T14:41:40.560327+0000","last_became_active":"2024-03-18T14:41:35.558687+0000","last_became_peered":"2024-03-18T14:41:35.558687+0000","last_unstale":"2024-03-18T14:41:40.560327+0000","last_undegraded":"2024-03-18T14:41:40.560327+0000","last_fullsized":"2024-03-18T14:41:40.560327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:25.399581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559780+0000","last_change":"2024-03-18T14:41:34.525754+0000","last_active":"2024-03-18T14:41:40.559780+0000","last_peered":"2024-03-18T14:41:40.559780+0000","last_clean":"2024-03-18T14:41:40.559780+0000","last_became_active":"2024-03-18T14:41:34.525453+0000","last_became_peered":"2024-03-18T14:41:34.525453+0000","last_unstale":"2024-03-18T14:41:40.559780+0000","last_undegraded":"2024-03-18T14:41:40.559780+0000","last_fullsized":"2024-03-18T14:41:40.559780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778594+0000","last_change":"2024-03-18T14:41:39.584128+0000","last_active":"2024-03-18T14:41:40.778594+0000","last_peered":"2024-03-18T14:41:40.778594+0000","last_clean":"2024-03-18T14:41:40.778594+0000","last_became_active":"2024-03-18T14:41:39.582472+0000","last_became_peered":"2024-03-18T14:41:39.582472+0000","last_unstale":"2024-03-18T14:41:40.778594+0000","last_undegraded":"2024-03-18T14:41:40.778594+0000","last_fullsized":"2024-03-18T14:41:40.778594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778049+0000","last_change":"2024-03-18T14:41:35.540356+0000","last_active":"2024-03-18T14:41:40.778049+0000","last_peered":"2024-03-18T14:41:40.778049+0000","last_clean":"2024-03-18T14:41:40.778049+0000","last_became_active":"2024-03-18T14:41:35.540060+0000","last_became_peered":"2024-03-18T14:41:35.540060+0000","last_unstale":"2024-03-18T14:41:40.778049+0000","last_undegraded":"2024-03-18T14:41:40.778049+0000","last_fullsized":"2024-03-18T14:41:40.778049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778106+0000","last_change":"2024-03-18T14:41:34.529076+0000","last_active":"2024-03-18T14:41:40.778106+0000","last_peered":"2024-03-18T14:41:40.778106+0000","last_clean":"2024-03-18T14:41:40.778106+0000","last_became_active":"2024-03-18T14:41:34.528794+0000","last_became_peered":"2024-03-18T14:41:34.528794+0000","last_unstale":"2024-03-18T14:41:40.778106+0000","last_undegraded":"2024-03-18T14:41:40.778106+0000","last_fullsized":"2024-03-18T14:41:40.778106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779855+0000","last_change":"2024-03-18T14:41:39.581385+0000","last_active":"2024-03-18T14:41:40.779855+0000","last_peered":"2024-03-18T14:41:40.779855+0000","last_clean":"2024-03-18T14:41:40.779855+0000","last_became_active":"2024-03-18T14:41:39.580972+0000","last_became_peered":"2024-03-18T14:41:39.580972+0000","last_unstale":"2024-03-18T14:41:40.779855+0000","last_undegraded":"2024-03-18T14:41:40.779855+0000","last_fullsized":"2024-03-18T14:41:40.779855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778334+0000","last_change":"2024-03-18T14:41:35.552692+0000","last_active":"2024-03-18T14:41:40.778334+0000","last_peered":"2024-03-18T14:41:40.778334+0000","last_clean":"2024-03-18T14:41:40.778334+0000","last_became_active":"2024-03-18T14:41:35.552340+0000","last_became_peered":"2024-03-18T14:41:35.552340+0000","last_unstale":"2024-03-18T14:41:40.778334+0000","last_undegraded":"2024-03-18T14:41:40.778334+0000","last_fullsized":"2024-03-18T14:41:40.778334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778015+0000","last_change":"2024-03-18T14:41:34.531973+0000","last_active":"2024-03-18T14:41:40.778015+0000","last_peered":"2024-03-18T14:41:40.778015+0000","last_clean":"2024-03-18T14:41:40.778015+0000","last_became_active":"2024-03-18T14:41:34.531796+0000","last_became_peered":"2024-03-18T14:41:34.531796+0000","last_unstale":"2024-03-18T14:41:40.778015+0000","last_undegraded":"2024-03-18T14:41:40.778015+0000","last_fullsized":"2024-03-18T14:41:40.778015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778621+0000","last_change":"2024-03-18T14:41:39.601287+0000","last_active":"2024-03-18T14:41:40.778621+0000","last_peered":"2024-03-18T14:41:40.778621+0000","last_clean":"2024-03-18T14:41:40.778621+0000","last_became_active":"2024-03-18T14:41:39.600855+0000","last_became_peered":"2024-03-18T14:41:39.600855+0000","last_unstale":"2024-03-18T14:41:40.778621+0000","last_undegraded":"2024-03-18T14:41:40.778621+0000","last_fullsized":"2024-03-18T14:41:40.778621+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563301+0000","last_change":"2024-03-18T14:41:35.547848+0000","last_active":"2024-03-18T14:41:40.563301+0000","last_peered":"2024-03-18T14:41:40.563301+0000","last_clean":"2024-03-18T14:41:40.563301+0000","last_became_active":"2024-03-18T14:41:35.547437+0000","last_became_peered":"2024-03-18T14:41:35.547437+0000","last_unstale":"2024-03-18T14:41:40.563301+0000","last_undegraded":"2024-03-18T14:41:40.563301+0000","last_fullsized":"2024-03-18T14:41:40.563301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559408+0000","last_change":"2024-03-18T14:41:34.528039+0000","last_active":"2024-03-18T14:41:40.559408+0000","last_peered":"2024-03-18T14:41:40.559408+0000","last_clean":"2024-03-18T14:41:40.559408+0000","last_became_active":"2024-03-18T14:41:34.527756+0000","last_became_peered":"2024-03-18T14:41:34.527756+0000","last_unstale":"2024-03-18T14:41:40.559408+0000","last_undegraded":"2024-03-18T14:41:40.559408+0000","last_fullsized":"2024-03-18T14:41:40.559408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778101+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:41:40.778101+0000","last_peered":"2024-03-18T14:41:40.778101+0000","last_clean":"2024-03-18T14:41:40.778101+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:41:40.778101+0000","last_undegraded":"2024-03-18T14:41:40.778101+0000","last_fullsized":"2024-03-18T14:41:40.778101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778699+0000","last_change":"2024-03-18T14:41:39.602108+0000","last_active":"2024-03-18T14:41:40.778699+0000","last_peered":"2024-03-18T14:41:40.778699+0000","last_clean":"2024-03-18T14:41:40.778699+0000","last_became_active":"2024-03-18T14:41:39.601732+0000","last_became_peered":"2024-03-18T14:41:39.601732+0000","last_unstale":"2024-03-18T14:41:40.778699+0000","last_undegraded":"2024-03-18T14:41:40.778699+0000","last_fullsized":"2024-03-18T14:41:40.778699+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778841+0000","last_change":"2024-03-18T14:41:35.544776+0000","last_active":"2024-03-18T14:41:40.778841+0000","last_peered":"2024-03-18T14:41:40.778841+0000","last_clean":"2024-03-18T14:41:40.778841+0000","last_became_active":"2024-03-18T14:41:35.544498+0000","last_became_peered":"2024-03-18T14:41:35.544498+0000","last_unstale":"2024-03-18T14:41:40.778841+0000","last_undegraded":"2024-03-18T14:41:40.778841+0000","last_fullsized":"2024-03-18T14:41:40.778841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780098+0000","last_change":"2024-03-18T14:41:34.531645+0000","last_active":"2024-03-18T14:41:40.780098+0000","last_peered":"2024-03-18T14:41:40.780098+0000","last_clean":"2024-03-18T14:41:40.780098+0000","last_became_active":"2024-03-18T14:41:34.530361+0000","last_became_peered":"2024-03-18T14:41:34.530361+0000","last_unstale":"2024-03-18T14:41:40.780098+0000","last_undegraded":"2024-03-18T14:41:40.780098+0000","last_fullsized":"2024-03-18T14:41:40.780098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778063+0000","last_change":"2024-03-18T14:41:39.601190+0000","last_active":"2024-03-18T14:41:40.778063+0000","last_peered":"2024-03-18T14:41:40.778063+0000","last_clean":"2024-03-18T14:41:40.778063+0000","last_became_active":"2024-03-18T14:41:39.600809+0000","last_became_peered":"2024-03-18T14:41:39.600809+0000","last_unstale":"2024-03-18T14:41:40.778063+0000","last_undegraded":"2024-03-18T14:41:40.778063+0000","last_fullsized":"2024-03-18T14:41:40.778063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559630+0000","last_change":"2024-03-18T14:41:35.549443+0000","last_active":"2024-03-18T14:41:40.559630+0000","last_peered":"2024-03-18T14:41:40.559630+0000","last_clean":"2024-03-18T14:41:40.559630+0000","last_became_active":"2024-03-18T14:41:35.546376+0000","last_became_peered":"2024-03-18T14:41:35.546376+0000","last_unstale":"2024-03-18T14:41:40.559630+0000","last_undegraded":"2024-03-18T14:41:40.559630+0000","last_fullsized":"2024-03-18T14:41:40.559630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779349+0000","last_change":"2024-03-18T14:41:34.528408+0000","last_active":"2024-03-18T14:41:40.779349+0000","last_peered":"2024-03-18T14:41:40.779349+0000","last_clean":"2024-03-18T14:41:40.779349+0000","last_became_active":"2024-03-18T14:41:34.527981+0000","last_became_peered":"2024-03-18T14:41:34.527981+0000","last_unstale":"2024-03-18T14:41:40.779349+0000","last_undegraded":"2024-03-18T14:41:40.779349+0000","last_fullsized":"2024-03-18T14:41:40.779349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778976+0000","last_change":"2024-03-18T14:41:39.586214+0000","last_active":"2024-03-18T14:41:40.778976+0000","last_peered":"2024-03-18T14:41:40.778976+0000","last_clean":"2024-03-18T14:41:40.778976+0000","last_became_active":"2024-03-18T14:41:39.585773+0000","last_became_peered":"2024-03-18T14:41:39.585773+0000","last_unstale":"2024-03-18T14:41:40.778976+0000","last_undegraded":"2024-03-18T14:41:40.778976+0000","last_fullsized":"2024-03-18T14:41:40.778976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:52:45.131607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778850+0000","last_change":"2024-03-18T14:41:35.553371+0000","last_active":"2024-03-18T14:41:40.778850+0000","last_peered":"2024-03-18T14:41:40.778850+0000","last_clean":"2024-03-18T14:41:40.778850+0000","last_became_active":"2024-03-18T14:41:35.552906+0000","last_became_peered":"2024-03-18T14:41:35.552906+0000","last_unstale":"2024-03-18T14:41:40.778850+0000","last_undegraded":"2024-03-18T14:41:40.778850+0000","last_fullsized":"2024-03-18T14:41:40.778850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778853+0000","last_change":"2024-03-18T14:41:34.532843+0000","last_active":"2024-03-18T14:41:40.778853+0000","last_peered":"2024-03-18T14:41:40.778853+0000","last_clean":"2024-03-18T14:41:40.778853+0000","last_became_active":"2024-03-18T14:41:34.532183+0000","last_became_peered":"2024-03-18T14:41:34.532183+0000","last_unstale":"2024-03-18T14:41:40.778853+0000","last_undegraded":"2024-03-18T14:41:40.778853+0000","last_fullsized":"2024-03-18T14:41:40.778853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559016+0000","last_change":"2024-03-18T14:41:39.603278+0000","last_active":"2024-03-18T14:41:40.559016+0000","last_peered":"2024-03-18T14:41:40.559016+0000","last_clean":"2024-03-18T14:41:40.559016+0000","last_became_active":"2024-03-18T14:41:39.602927+0000","last_became_peered":"2024-03-18T14:41:39.602927+0000","last_unstale":"2024-03-18T14:41:40.559016+0000","last_undegraded":"2024-03-18T14:41:40.559016+0000","last_fullsized":"2024-03-18T14:41:40.559016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778747+0000","last_change":"2024-03-18T14:41:35.552924+0000","last_active":"2024-03-18T14:41:40.778747+0000","last_peered":"2024-03-18T14:41:40.778747+0000","last_clean":"2024-03-18T14:41:40.778747+0000","last_became_active":"2024-03-18T14:41:35.552514+0000","last_became_peered":"2024-03-18T14:41:35.552514+0000","last_unstale":"2024-03-18T14:41:40.778747+0000","last_undegraded":"2024-03-18T14:41:40.778747+0000","last_fullsized":"2024-03-18T14:41:40.778747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560464+0000","last_change":"2024-03-18T14:41:34.534565+0000","last_active":"2024-03-18T14:41:40.560464+0000","last_peered":"2024-03-18T14:41:40.560464+0000","last_clean":"2024-03-18T14:41:40.560464+0000","last_became_active":"2024-03-18T14:41:34.534302+0000","last_became_peered":"2024-03-18T14:41:34.534302+0000","last_unstale":"2024-03-18T14:41:40.560464+0000","last_undegraded":"2024-03-18T14:41:40.560464+0000","last_fullsized":"2024-03-18T14:41:40.560464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559373+0000","last_change":"2024-03-18T14:41:39.580698+0000","last_active":"2024-03-18T14:41:40.559373+0000","last_peered":"2024-03-18T14:41:40.559373+0000","last_clean":"2024-03-18T14:41:40.559373+0000","last_became_active":"2024-03-18T14:41:39.580161+0000","last_became_peered":"2024-03-18T14:41:39.580161+0000","last_unstale":"2024-03-18T14:41:40.559373+0000","last_undegraded":"2024-03-18T14:41:40.559373+0000","last_fullsized":"2024-03-18T14:41:40.559373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:03.265239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778352+0000","last_change":"2024-03-18T14:41:35.547876+0000","last_active":"2024-03-18T14:41:40.778352+0000","last_peered":"2024-03-18T14:41:40.778352+0000","last_clean":"2024-03-18T14:41:40.778352+0000","last_became_active":"2024-03-18T14:41:35.547615+0000","last_became_peered":"2024-03-18T14:41:35.547615+0000","last_unstale":"2024-03-18T14:41:40.778352+0000","last_undegraded":"2024-03-18T14:41:40.778352+0000","last_fullsized":"2024-03-18T14:41:40.778352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777856+0000","last_change":"2024-03-18T14:41:34.526247+0000","last_active":"2024-03-18T14:41:40.777856+0000","last_peered":"2024-03-18T14:41:40.777856+0000","last_clean":"2024-03-18T14:41:40.777856+0000","last_became_active":"2024-03-18T14:41:34.525952+0000","last_became_peered":"2024-03-18T14:41:34.525952+0000","last_unstale":"2024-03-18T14:41:40.777856+0000","last_undegraded":"2024-03-18T14:41:40.777856+0000","last_fullsized":"2024-03-18T14:41:40.777856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:39.580945+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:39.580421+0000","last_became_peered":"2024-03-18T14:41:39.580421+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778016+0000","last_change":"2024-03-18T14:41:35.554425+0000","last_active":"2024-03-18T14:41:40.778016+0000","last_peered":"2024-03-18T14:41:40.778016+0000","last_clean":"2024-03-18T14:41:40.778016+0000","last_became_active":"2024-03-18T14:41:35.554164+0000","last_became_peered":"2024-03-18T14:41:35.554164+0000","last_unstale":"2024-03-18T14:41:40.778016+0000","last_undegraded":"2024-03-18T14:41:40.778016+0000","last_fullsized":"2024-03-18T14:41:40.778016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778159+0000","last_change":"2024-03-18T14:41:34.528726+0000","last_active":"2024-03-18T14:41:40.778159+0000","last_peered":"2024-03-18T14:41:40.778159+0000","last_clean":"2024-03-18T14:41:40.778159+0000","last_became_active":"2024-03-18T14:41:34.528424+0000","last_became_peered":"2024-03-18T14:41:34.528424+0000","last_unstale":"2024-03-18T14:41:40.778159+0000","last_undegraded":"2024-03-18T14:41:40.778159+0000","last_fullsized":"2024-03-18T14:41:40.778159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564629+0000","last_change":"2024-03-18T14:41:39.602221+0000","last_active":"2024-03-18T14:41:40.564629+0000","last_peered":"2024-03-18T14:41:40.564629+0000","last_clean":"2024-03-18T14:41:40.564629+0000","last_became_active":"2024-03-18T14:41:39.601652+0000","last_became_peered":"2024-03-18T14:41:39.601652+0000","last_unstale":"2024-03-18T14:41:40.564629+0000","last_undegraded":"2024-03-18T14:41:40.564629+0000","last_fullsized":"2024-03-18T14:41:40.564629+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:00.363235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778400+0000","last_change":"2024-03-18T14:41:35.548474+0000","last_active":"2024-03-18T14:41:40.778400+0000","last_peered":"2024-03-18T14:41:40.778400+0000","last_clean":"2024-03-18T14:41:40.778400+0000","last_became_active":"2024-03-18T14:41:35.548193+0000","last_became_peered":"2024-03-18T14:41:35.548193+0000","last_unstale":"2024-03-18T14:41:40.778400+0000","last_undegraded":"2024-03-18T14:41:40.778400+0000","last_fullsized":"2024-03-18T14:41:40.778400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559701+0000","last_change":"2024-03-18T14:41:34.529770+0000","last_active":"2024-03-18T14:41:40.559701+0000","last_peered":"2024-03-18T14:41:40.559701+0000","last_clean":"2024-03-18T14:41:40.559701+0000","last_became_active":"2024-03-18T14:41:34.529499+0000","last_became_peered":"2024-03-18T14:41:34.529499+0000","last_unstale":"2024-03-18T14:41:40.559701+0000","last_undegraded":"2024-03-18T14:41:40.559701+0000","last_fullsized":"2024-03-18T14:41:40.559701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564084+0000","last_change":"2024-03-18T14:41:39.601622+0000","last_active":"2024-03-18T14:41:40.564084+0000","last_peered":"2024-03-18T14:41:40.564084+0000","last_clean":"2024-03-18T14:41:40.564084+0000","last_became_active":"2024-03-18T14:41:39.601209+0000","last_became_peered":"2024-03-18T14:41:39.601209+0000","last_unstale":"2024-03-18T14:41:40.564084+0000","last_undegraded":"2024-03-18T14:41:40.564084+0000","last_fullsized":"2024-03-18T14:41:40.564084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:11.443644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778881+0000","last_change":"2024-03-18T14:41:35.547642+0000","last_active":"2024-03-18T14:41:40.778881+0000","last_peered":"2024-03-18T14:41:40.778881+0000","last_clean":"2024-03-18T14:41:40.778881+0000","last_became_active":"2024-03-18T14:41:35.547354+0000","last_became_peered":"2024-03-18T14:41:35.547354+0000","last_unstale":"2024-03-18T14:41:40.778881+0000","last_undegraded":"2024-03-18T14:41:40.778881+0000","last_fullsized":"2024-03-18T14:41:40.778881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779200+0000","last_change":"2024-03-18T14:41:34.533998+0000","last_active":"2024-03-18T14:41:40.779200+0000","last_peered":"2024-03-18T14:41:40.779200+0000","last_clean":"2024-03-18T14:41:40.779200+0000","last_became_active":"2024-03-18T14:41:34.533597+0000","last_became_peered":"2024-03-18T14:41:34.533597+0000","last_unstale":"2024-03-18T14:41:40.779200+0000","last_undegraded":"2024-03-18T14:41:40.779200+0000","last_fullsized":"2024-03-18T14:41:40.779200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560211+0000","last_change":"2024-03-18T14:41:39.601126+0000","last_active":"2024-03-18T14:41:40.560211+0000","last_peered":"2024-03-18T14:41:40.560211+0000","last_clean":"2024-03-18T14:41:40.560211+0000","last_became_active":"2024-03-18T14:41:39.600751+0000","last_became_peered":"2024-03-18T14:41:39.600751+0000","last_unstale":"2024-03-18T14:41:40.560211+0000","last_undegraded":"2024-03-18T14:41:40.560211+0000","last_fullsized":"2024-03-18T14:41:40.560211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778816+0000","last_change":"2024-03-18T14:41:35.550195+0000","last_active":"2024-03-18T14:41:40.778816+0000","last_peered":"2024-03-18T14:41:40.778816+0000","last_clean":"2024-03-18T14:41:40.778816+0000","last_became_active":"2024-03-18T14:41:35.549894+0000","last_became_peered":"2024-03-18T14:41:35.549894+0000","last_unstale":"2024-03-18T14:41:40.778816+0000","last_undegraded":"2024-03-18T14:41:40.778816+0000","last_fullsized":"2024-03-18T14:41:40.778816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:45.045404+0000","last_change":"2024-03-18T14:41:34.536841+0000","last_active":"2024-03-18T14:41:45.045404+0000","last_peered":"2024-03-18T14:41:45.045404+0000","last_clean":"2024-03-18T14:41:45.045404+0000","last_became_active":"2024-03-18T14:41:34.536536+0000","last_became_peered":"2024-03-18T14:41:34.536536+0000","last_unstale":"2024-03-18T14:41:45.045404+0000","last_undegraded":"2024-03-18T14:41:45.045404+0000","last_fullsized":"2024-03-18T14:41:45.045404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779808+0000","last_change":"2024-03-18T14:41:39.582692+0000","last_active":"2024-03-18T14:41:40.779808+0000","last_peered":"2024-03-18T14:41:40.779808+0000","last_clean":"2024-03-18T14:41:40.779808+0000","last_became_active":"2024-03-18T14:41:39.582285+0000","last_became_peered":"2024-03-18T14:41:39.582285+0000","last_unstale":"2024-03-18T14:41:40.779808+0000","last_undegraded":"2024-03-18T14:41:40.779808+0000","last_fullsized":"2024-03-18T14:41:40.779808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559959+0000","last_change":"2024-03-18T14:41:35.547692+0000","last_active":"2024-03-18T14:41:40.559959+0000","last_peered":"2024-03-18T14:41:40.559959+0000","last_clean":"2024-03-18T14:41:40.559959+0000","last_became_active":"2024-03-18T14:41:35.547381+0000","last_became_peered":"2024-03-18T14:41:35.547381+0000","last_unstale":"2024-03-18T14:41:40.559959+0000","last_undegraded":"2024-03-18T14:41:40.559959+0000","last_fullsized":"2024-03-18T14:41:40.559959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:13.037877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779251+0000","last_change":"2024-03-18T14:41:34.529622+0000","last_active":"2024-03-18T14:41:40.779251+0000","last_peered":"2024-03-18T14:41:40.779251+0000","last_clean":"2024-03-18T14:41:40.779251+0000","last_became_active":"2024-03-18T14:41:34.528752+0000","last_became_peered":"2024-03-18T14:41:34.528752+0000","last_unstale":"2024-03-18T14:41:40.779251+0000","last_undegraded":"2024-03-18T14:41:40.779251+0000","last_fullsized":"2024-03-18T14:41:40.779251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560121+0000","last_change":"2024-03-18T14:41:39.585230+0000","last_active":"2024-03-18T14:41:40.560121+0000","last_peered":"2024-03-18T14:41:40.560121+0000","last_clean":"2024-03-18T14:41:40.560121+0000","last_became_active":"2024-03-18T14:41:39.584896+0000","last_became_peered":"2024-03-18T14:41:39.584896+0000","last_unstale":"2024-03-18T14:41:40.560121+0000","last_undegraded":"2024-03-18T14:41:40.560121+0000","last_fullsized":"2024-03-18T14:41:40.560121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:35.552508+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:35.552221+0000","last_became_peered":"2024-03-18T14:41:35.552221+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560164+0000","last_change":"2024-03-18T14:41:34.534226+0000","last_active":"2024-03-18T14:41:40.560164+0000","last_peered":"2024-03-18T14:41:40.560164+0000","last_clean":"2024-03-18T14:41:40.560164+0000","last_became_active":"2024-03-18T14:41:34.533977+0000","last_became_peered":"2024-03-18T14:41:34.533977+0000","last_unstale":"2024-03-18T14:41:40.560164+0000","last_undegraded":"2024-03-18T14:41:40.560164+0000","last_fullsized":"2024-03-18T14:41:40.560164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563415+0000","last_change":"2024-03-18T14:41:39.583657+0000","last_active":"2024-03-18T14:41:40.563415+0000","last_peered":"2024-03-18T14:41:40.563415+0000","last_clean":"2024-03-18T14:41:40.563415+0000","last_became_active":"2024-03-18T14:41:39.583280+0000","last_became_peered":"2024-03-18T14:41:39.583280+0000","last_unstale":"2024-03-18T14:41:40.563415+0000","last_undegraded":"2024-03-18T14:41:40.563415+0000","last_fullsized":"2024-03-18T14:41:40.563415+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560126+0000","last_change":"2024-03-18T14:41:35.546368+0000","last_active":"2024-03-18T14:41:40.560126+0000","last_peered":"2024-03-18T14:41:40.560126+0000","last_clean":"2024-03-18T14:41:40.560126+0000","last_became_active":"2024-03-18T14:41:35.546101+0000","last_became_peered":"2024-03-18T14:41:35.546101+0000","last_unstale":"2024-03-18T14:41:40.560126+0000","last_undegraded":"2024-03-18T14:41:40.560126+0000","last_fullsized":"2024-03-18T14:41:40.560126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779013+0000","last_change":"2024-03-18T14:41:34.535621+0000","last_active":"2024-03-18T14:41:40.779013+0000","last_peered":"2024-03-18T14:41:40.779013+0000","last_clean":"2024-03-18T14:41:40.779013+0000","last_became_active":"2024-03-18T14:41:34.535351+0000","last_became_peered":"2024-03-18T14:41:34.535351+0000","last_unstale":"2024-03-18T14:41:40.779013+0000","last_undegraded":"2024-03-18T14:41:40.779013+0000","last_fullsized":"2024-03-18T14:41:40.779013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563431+0000","last_change":"2024-03-18T14:41:39.602374+0000","last_active":"2024-03-18T14:41:40.563431+0000","last_peered":"2024-03-18T14:41:40.563431+0000","last_clean":"2024-03-18T14:41:40.563431+0000","last_became_active":"2024-03-18T14:41:39.600913+0000","last_became_peered":"2024-03-18T14:41:39.600913+0000","last_unstale":"2024-03-18T14:41:40.563431+0000","last_undegraded":"2024-03-18T14:41:40.563431+0000","last_fullsized":"2024-03-18T14:41:40.563431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778290+0000","last_change":"2024-03-18T14:41:39.602659+0000","last_active":"2024-03-18T14:41:40.778290+0000","last_peered":"2024-03-18T14:41:40.778290+0000","last_clean":"2024-03-18T14:41:40.778290+0000","last_became_active":"2024-03-18T14:41:39.602286+0000","last_became_peered":"2024-03-18T14:41:39.602286+0000","last_unstale":"2024-03-18T14:41:40.778290+0000","last_undegraded":"2024-03-18T14:41:40.778290+0000","last_fullsized":"2024-03-18T14:41:40.778290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778038+0000","last_change":"2024-03-18T14:41:35.545020+0000","last_active":"2024-03-18T14:41:40.778038+0000","last_peered":"2024-03-18T14:41:40.778038+0000","last_clean":"2024-03-18T14:41:40.778038+0000","last_became_active":"2024-03-18T14:41:35.544697+0000","last_became_peered":"2024-03-18T14:41:35.544697+0000","last_unstale":"2024-03-18T14:41:40.778038+0000","last_undegraded":"2024-03-18T14:41:40.778038+0000","last_fullsized":"2024-03-18T14:41:40.778038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559154+0000","last_change":"2024-03-18T14:41:34.527624+0000","last_active":"2024-03-18T14:41:40.559154+0000","last_peered":"2024-03-18T14:41:40.559154+0000","last_clean":"2024-03-18T14:41:40.559154+0000","last_became_active":"2024-03-18T14:41:34.527291+0000","last_became_peered":"2024-03-18T14:41:34.527291+0000","last_unstale":"2024-03-18T14:41:40.559154+0000","last_undegraded":"2024-03-18T14:41:40.559154+0000","last_fullsized":"2024-03-18T14:41:40.559154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779046+0000","last_change":"2024-03-18T14:41:39.575946+0000","last_active":"2024-03-18T14:41:40.779046+0000","last_peered":"2024-03-18T14:41:40.779046+0000","last_clean":"2024-03-18T14:41:40.779046+0000","last_became_active":"2024-03-18T14:41:39.575618+0000","last_became_peered":"2024-03-18T14:41:39.575618+0000","last_unstale":"2024-03-18T14:41:40.779046+0000","last_undegraded":"2024-03-18T14:41:40.779046+0000","last_fullsized":"2024-03-18T14:41:40.779046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563258+0000","last_change":"2024-03-18T14:41:35.548573+0000","last_active":"2024-03-18T14:41:40.563258+0000","last_peered":"2024-03-18T14:41:40.563258+0000","last_clean":"2024-03-18T14:41:40.563258+0000","last_became_active":"2024-03-18T14:41:35.548284+0000","last_became_peered":"2024-03-18T14:41:35.548284+0000","last_unstale":"2024-03-18T14:41:40.563258+0000","last_undegraded":"2024-03-18T14:41:40.563258+0000","last_fullsized":"2024-03-18T14:41:40.563258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563616+0000","last_change":"2024-03-18T14:41:34.536158+0000","last_active":"2024-03-18T14:41:40.563616+0000","last_peered":"2024-03-18T14:41:40.563616+0000","last_clean":"2024-03-18T14:41:40.563616+0000","last_became_active":"2024-03-18T14:41:34.535879+0000","last_became_peered":"2024-03-18T14:41:34.535879+0000","last_unstale":"2024-03-18T14:41:40.563616+0000","last_undegraded":"2024-03-18T14:41:40.563616+0000","last_fullsized":"2024-03-18T14:41:40.563616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778365+0000","last_change":"2024-03-18T14:41:34.528073+0000","last_active":"2024-03-18T14:41:40.778365+0000","last_peered":"2024-03-18T14:41:40.778365+0000","last_clean":"2024-03-18T14:41:40.778365+0000","last_became_active":"2024-03-18T14:41:34.527245+0000","last_became_peered":"2024-03-18T14:41:34.527245+0000","last_unstale":"2024-03-18T14:41:40.778365+0000","last_undegraded":"2024-03-18T14:41:40.778365+0000","last_fullsized":"2024-03-18T14:41:40.778365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779106+0000","last_change":"2024-03-18T14:41:35.560334+0000","last_active":"2024-03-18T14:41:40.779106+0000","last_peered":"2024-03-18T14:41:40.779106+0000","last_clean":"2024-03-18T14:41:40.779106+0000","last_became_active":"2024-03-18T14:41:35.559938+0000","last_became_peered":"2024-03-18T14:41:35.559938+0000","last_unstale":"2024-03-18T14:41:40.779106+0000","last_undegraded":"2024-03-18T14:41:40.779106+0000","last_fullsized":"2024-03-18T14:41:40.779106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780010+0000","last_change":"2024-03-18T14:41:39.578410+0000","last_active":"2024-03-18T14:41:40.780010+0000","last_peered":"2024-03-18T14:41:40.780010+0000","last_clean":"2024-03-18T14:41:40.780010+0000","last_became_active":"2024-03-18T14:41:39.577919+0000","last_became_peered":"2024-03-18T14:41:39.577919+0000","last_unstale":"2024-03-18T14:41:40.780010+0000","last_undegraded":"2024-03-18T14:41:40.780010+0000","last_fullsized":"2024-03-18T14:41:40.780010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:59.569962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563940+0000","last_change":"2024-03-18T14:41:34.535233+0000","last_active":"2024-03-18T14:41:40.563940+0000","last_peered":"2024-03-18T14:41:40.563940+0000","last_clean":"2024-03-18T14:41:40.563940+0000","last_became_active":"2024-03-18T14:41:34.534879+0000","last_became_peered":"2024-03-18T14:41:34.534879+0000","last_unstale":"2024-03-18T14:41:40.563940+0000","last_undegraded":"2024-03-18T14:41:40.563940+0000","last_fullsized":"2024-03-18T14:41:40.563940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559603+0000","last_change":"2024-03-18T14:41:35.535546+0000","last_active":"2024-03-18T14:41:40.559603+0000","last_peered":"2024-03-18T14:41:40.559603+0000","last_clean":"2024-03-18T14:41:40.559603+0000","last_became_active":"2024-03-18T14:41:35.535269+0000","last_became_peered":"2024-03-18T14:41:35.535269+0000","last_unstale":"2024-03-18T14:41:40.559603+0000","last_undegraded":"2024-03-18T14:41:40.559603+0000","last_fullsized":"2024-03-18T14:41:40.559603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:39.583244+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:39.582882+0000","last_became_peered":"2024-03-18T14:41:39.582882+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:10:48.564938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777919+0000","last_change":"2024-03-18T14:41:35.554708+0000","last_active":"2024-03-18T14:41:40.777919+0000","last_peered":"2024-03-18T14:41:40.777919+0000","last_clean":"2024-03-18T14:41:40.777919+0000","last_became_active":"2024-03-18T14:41:35.554411+0000","last_became_peered":"2024-03-18T14:41:35.554411+0000","last_unstale":"2024-03-18T14:41:40.777919+0000","last_undegraded":"2024-03-18T14:41:40.777919+0000","last_fullsized":"2024-03-18T14:41:40.777919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777933+0000","last_change":"2024-03-18T14:41:35.543401+0000","last_active":"2024-03-18T14:41:40.777933+0000","last_peered":"2024-03-18T14:41:40.777933+0000","last_clean":"2024-03-18T14:41:40.777933+0000","last_became_active":"2024-03-18T14:41:35.543034+0000","last_became_peered":"2024-03-18T14:41:35.543034+0000","last_unstale":"2024-03-18T14:41:40.777933+0000","last_undegraded":"2024-03-18T14:41:40.777933+0000","last_fullsized":"2024-03-18T14:41:40.777933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559616+0000","last_change":"2024-03-18T14:41:35.551162+0000","last_active":"2024-03-18T14:41:40.559616+0000","last_peered":"2024-03-18T14:41:40.559616+0000","last_clean":"2024-03-18T14:41:40.559616+0000","last_became_active":"2024-03-18T14:41:35.550839+0000","last_became_peered":"2024-03-18T14:41:35.550839+0000","last_unstale":"2024-03-18T14:41:40.559616+0000","last_undegraded":"2024-03-18T14:41:40.559616+0000","last_fullsized":"2024-03-18T14:41:40.559616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779023+0000","last_change":"2024-03-18T14:41:35.554787+0000","last_active":"2024-03-18T14:41:40.779023+0000","last_peered":"2024-03-18T14:41:40.779023+0000","last_clean":"2024-03-18T14:41:40.779023+0000","last_became_active":"2024-03-18T14:41:35.554484+0000","last_became_peered":"2024-03-18T14:41:35.554484+0000","last_unstale":"2024-03-18T14:41:40.779023+0000","last_undegraded":"2024-03-18T14:41:40.779023+0000","last_fullsized":"2024-03-18T14:41:40.779023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564088+0000","last_change":"2024-03-18T14:41:35.544958+0000","last_active":"2024-03-18T14:41:40.564088+0000","last_peered":"2024-03-18T14:41:40.564088+0000","last_clean":"2024-03-18T14:41:40.564088+0000","last_became_active":"2024-03-18T14:41:35.544600+0000","last_became_peered":"2024-03-18T14:41:35.544600+0000","last_unstale":"2024-03-18T14:41:40.564088+0000","last_undegraded":"2024-03-18T14:41:40.564088+0000","last_fullsized":"2024-03-18T14:41:40.564088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778046+0000","last_change":"2024-03-18T14:41:35.549309+0000","last_active":"2024-03-18T14:41:40.778046+0000","last_peered":"2024-03-18T14:41:40.778046+0000","last_clean":"2024-03-18T14:41:40.778046+0000","last_became_active":"2024-03-18T14:41:35.549039+0000","last_became_peered":"2024-03-18T14:41:35.549039+0000","last_unstale":"2024-03-18T14:41:40.778046+0000","last_undegraded":"2024-03-18T14:41:40.778046+0000","last_fullsized":"2024-03-18T14:41:40.778046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778908+0000","last_change":"2024-03-18T14:41:35.563387+0000","last_active":"2024-03-18T14:41:40.778908+0000","last_peered":"2024-03-18T14:41:40.778908+0000","last_clean":"2024-03-18T14:41:40.778908+0000","last_became_active":"2024-03-18T14:41:35.562621+0000","last_became_peered":"2024-03-18T14:41:35.562621+0000","last_unstale":"2024-03-18T14:41:40.778908+0000","last_undegraded":"2024-03-18T14:41:40.778908+0000","last_fullsized":"2024-03-18T14:41:40.778908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564514+0000","last_change":"2024-03-18T14:41:35.549988+0000","last_active":"2024-03-18T14:41:40.564514+0000","last_peered":"2024-03-18T14:41:40.564514+0000","last_clean":"2024-03-18T14:41:40.564514+0000","last_became_active":"2024-03-18T14:41:35.549713+0000","last_became_peered":"2024-03-18T14:41:35.549713+0000","last_unstale":"2024-03-18T14:41:40.564514+0000","last_undegraded":"2024-03-18T14:41:40.564514+0000","last_fullsized":"2024-03-18T14:41:40.564514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559089+0000","last_change":"2024-03-18T14:41:35.540097+0000","last_active":"2024-03-18T14:41:40.559089+0000","last_peered":"2024-03-18T14:41:40.559089+0000","last_clean":"2024-03-18T14:41:40.559089+0000","last_became_active":"2024-03-18T14:41:35.539823+0000","last_became_peered":"2024-03-18T14:41:35.539823+0000","last_unstale":"2024-03-18T14:41:40.559089+0000","last_undegraded":"2024-03-18T14:41:40.559089+0000","last_fullsized":"2024-03-18T14:41:40.559089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563495+0000","last_change":"2024-03-18T14:41:35.544547+0000","last_active":"2024-03-18T14:41:40.563495+0000","last_peered":"2024-03-18T14:41:40.563495+0000","last_clean":"2024-03-18T14:41:40.563495+0000","last_became_active":"2024-03-18T14:41:35.544026+0000","last_became_peered":"2024-03-18T14:41:35.544026+0000","last_unstale":"2024-03-18T14:41:40.563495+0000","last_undegraded":"2024-03-18T14:41:40.563495+0000","last_fullsized":"2024-03-18T14:41:40.563495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559538+0000","last_change":"2024-03-18T14:41:35.545773+0000","last_active":"2024-03-18T14:41:40.559538+0000","last_peered":"2024-03-18T14:41:40.559538+0000","last_clean":"2024-03-18T14:41:40.559538+0000","last_became_active":"2024-03-18T14:41:35.545449+0000","last_became_peered":"2024-03-18T14:41:35.545449+0000","last_unstale":"2024-03-18T14:41:40.559538+0000","last_undegraded":"2024-03-18T14:41:40.559538+0000","last_fullsized":"2024-03-18T14:41:40.559538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778786+0000","last_change":"2024-03-18T14:41:35.563339+0000","last_active":"2024-03-18T14:41:40.778786+0000","last_peered":"2024-03-18T14:41:40.778786+0000","last_clean":"2024-03-18T14:41:40.778786+0000","last_became_active":"2024-03-18T14:41:35.563117+0000","last_became_peered":"2024-03-18T14:41:35.563117+0000","last_unstale":"2024-03-18T14:41:40.778786+0000","last_undegraded":"2024-03-18T14:41:40.778786+0000","last_fullsized":"2024-03-18T14:41:40.778786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778522+0000","last_change":"2024-03-18T14:41:35.559014+0000","last_active":"2024-03-18T14:41:40.778522+0000","last_peered":"2024-03-18T14:41:40.778522+0000","last_clean":"2024-03-18T14:41:40.778522+0000","last_became_active":"2024-03-18T14:41:35.558349+0000","last_became_peered":"2024-03-18T14:41:35.558349+0000","last_unstale":"2024-03-18T14:41:40.778522+0000","last_undegraded":"2024-03-18T14:41:40.778522+0000","last_fullsized":"2024-03-18T14:41:40.778522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778971+0000","last_change":"2024-03-18T14:41:35.545841+0000","last_active":"2024-03-18T14:41:40.778971+0000","last_peered":"2024-03-18T14:41:40.778971+0000","last_clean":"2024-03-18T14:41:40.778971+0000","last_became_active":"2024-03-18T14:41:35.545543+0000","last_became_peered":"2024-03-18T14:41:35.545543+0000","last_unstale":"2024-03-18T14:41:40.778971+0000","last_undegraded":"2024-03-18T14:41:40.778971+0000","last_fullsized":"2024-03-18T14:41:40.778971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778782+0000","last_change":"2024-03-18T14:41:35.551018+0000","last_active":"2024-03-18T14:41:40.778782+0000","last_peered":"2024-03-18T14:41:40.778782+0000","last_clean":"2024-03-18T14:41:40.778782+0000","last_became_active":"2024-03-18T14:41:35.550737+0000","last_became_peered":"2024-03-18T14:41:35.550737+0000","last_unstale":"2024-03-18T14:41:40.778782+0000","last_undegraded":"2024-03-18T14:41:40.778782+0000","last_fullsized":"2024-03-18T14:41:40.778782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560481+0000","last_change":"2024-03-18T14:41:35.547133+0000","last_active":"2024-03-18T14:41:40.560481+0000","last_peered":"2024-03-18T14:41:40.560481+0000","last_clean":"2024-03-18T14:41:40.560481+0000","last_became_active":"2024-03-18T14:41:35.546778+0000","last_became_peered":"2024-03-18T14:41:35.546778+0000","last_unstale":"2024-03-18T14:41:40.560481+0000","last_undegraded":"2024-03-18T14:41:40.560481+0000","last_fullsized":"2024-03-18T14:41:40.560481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559067+0000","last_change":"2024-03-18T14:41:35.550366+0000","last_active":"2024-03-18T14:41:40.559067+0000","last_peered":"2024-03-18T14:41:40.559067+0000","last_clean":"2024-03-18T14:41:40.559067+0000","last_became_active":"2024-03-18T14:41:35.550111+0000","last_became_peered":"2024-03-18T14:41:35.550111+0000","last_unstale":"2024-03-18T14:41:40.559067+0000","last_undegraded":"2024-03-18T14:41:40.559067+0000","last_fullsized":"2024-03-18T14:41:40.559067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778058+0000","last_change":"2024-03-18T14:41:35.536085+0000","last_active":"2024-03-18T14:41:40.778058+0000","last_peered":"2024-03-18T14:41:40.778058+0000","last_clean":"2024-03-18T14:41:40.778058+0000","last_became_active":"2024-03-18T14:41:35.535769+0000","last_became_peered":"2024-03-18T14:41:35.535769+0000","last_unstale":"2024-03-18T14:41:40.778058+0000","last_undegraded":"2024-03-18T14:41:40.778058+0000","last_fullsized":"2024-03-18T14:41:40.778058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778571+0000","last_change":"2024-03-18T14:41:35.550291+0000","last_active":"2024-03-18T14:41:40.778571+0000","last_peered":"2024-03-18T14:41:40.778571+0000","last_clean":"2024-03-18T14:41:40.778571+0000","last_became_active":"2024-03-18T14:41:35.549994+0000","last_became_peered":"2024-03-18T14:41:35.549994+0000","last_unstale":"2024-03-18T14:41:40.778571+0000","last_undegraded":"2024-03-18T14:41:40.778571+0000","last_fullsized":"2024-03-18T14:41:40.778571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:20.446494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563821+0000","last_change":"2024-03-18T14:41:35.551464+0000","last_active":"2024-03-18T14:41:40.563821+0000","last_peered":"2024-03-18T14:41:40.563821+0000","last_clean":"2024-03-18T14:41:40.563821+0000","last_became_active":"2024-03-18T14:41:35.551200+0000","last_became_peered":"2024-03-18T14:41:35.551200+0000","last_unstale":"2024-03-18T14:41:40.563821+0000","last_undegraded":"2024-03-18T14:41:40.563821+0000","last_fullsized":"2024-03-18T14:41:40.563821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559869+0000","last_change":"2024-03-18T14:41:35.558583+0000","last_active":"2024-03-18T14:41:40.559869+0000","last_peered":"2024-03-18T14:41:40.559869+0000","last_clean":"2024-03-18T14:41:40.559869+0000","last_became_active":"2024-03-18T14:41:35.557388+0000","last_became_peered":"2024-03-18T14:41:35.557388+0000","last_unstale":"2024-03-18T14:41:40.559869+0000","last_undegraded":"2024-03-18T14:41:40.559869+0000","last_fullsized":"2024-03-18T14:41:40.559869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778685+0000","last_change":"2024-03-18T14:41:35.556362+0000","last_active":"2024-03-18T14:41:40.778685+0000","last_peered":"2024-03-18T14:41:40.778685+0000","last_clean":"2024-03-18T14:41:40.778685+0000","last_became_active":"2024-03-18T14:41:35.556073+0000","last_became_peered":"2024-03-18T14:41:35.556073+0000","last_unstale":"2024-03-18T14:41:40.778685+0000","last_undegraded":"2024-03-18T14:41:40.778685+0000","last_fullsized":"2024-03-18T14:41:40.778685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778827+0000","last_change":"2024-03-18T14:41:35.553776+0000","last_active":"2024-03-18T14:41:40.778827+0000","last_peered":"2024-03-18T14:41:40.778827+0000","last_clean":"2024-03-18T14:41:40.778827+0000","last_became_active":"2024-03-18T14:41:35.553413+0000","last_became_peered":"2024-03-18T14:41:35.553413+0000","last_unstale":"2024-03-18T14:41:40.778827+0000","last_undegraded":"2024-03-18T14:41:40.778827+0000","last_fullsized":"2024-03-18T14:41:40.778827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778851+0000","last_change":"2024-03-18T14:41:35.556025+0000","last_active":"2024-03-18T14:41:40.778851+0000","last_peered":"2024-03-18T14:41:40.778851+0000","last_clean":"2024-03-18T14:41:40.778851+0000","last_became_active":"2024-03-18T14:41:35.555322+0000","last_became_peered":"2024-03-18T14:41:35.555322+0000","last_unstale":"2024-03-18T14:41:40.778851+0000","last_undegraded":"2024-03-18T14:41:40.778851+0000","last_fullsized":"2024-03-18T14:41:40.778851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563141+0000","last_change":"2024-03-18T14:41:35.545290+0000","last_active":"2024-03-18T14:41:40.563141+0000","last_peered":"2024-03-18T14:41:40.563141+0000","last_clean":"2024-03-18T14:41:40.563141+0000","last_became_active":"2024-03-18T14:41:35.543962+0000","last_became_peered":"2024-03-18T14:41:35.543962+0000","last_unstale":"2024-03-18T14:41:40.563141+0000","last_undegraded":"2024-03-18T14:41:40.563141+0000","last_fullsized":"2024-03-18T14:41:40.563141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778288+0000","last_change":"2024-03-18T14:41:35.556987+0000","last_active":"2024-03-18T14:41:40.778288+0000","last_peered":"2024-03-18T14:41:40.778288+0000","last_clean":"2024-03-18T14:41:40.778288+0000","last_became_active":"2024-03-18T14:41:35.556713+0000","last_became_peered":"2024-03-18T14:41:35.556713+0000","last_unstale":"2024-03-18T14:41:40.778288+0000","last_undegraded":"2024-03-18T14:41:40.778288+0000","last_fullsized":"2024-03-18T14:41:40.778288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559461+0000","last_change":"2024-03-18T14:41:35.548724+0000","last_active":"2024-03-18T14:41:40.559461+0000","last_peered":"2024-03-18T14:41:40.559461+0000","last_clean":"2024-03-18T14:41:40.559461+0000","last_became_active":"2024-03-18T14:41:35.547693+0000","last_became_peered":"2024-03-18T14:41:35.547693+0000","last_unstale":"2024-03-18T14:41:40.559461+0000","last_undegraded":"2024-03-18T14:41:40.559461+0000","last_fullsized":"2024-03-18T14:41:40.559461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:35.544193+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:35.543909+0000","last_became_peered":"2024-03-18T14:41:35.543909+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559445+0000","last_change":"2024-03-18T14:41:35.549929+0000","last_active":"2024-03-18T14:41:40.559445+0000","last_peered":"2024-03-18T14:41:40.559445+0000","last_clean":"2024-03-18T14:41:40.559445+0000","last_became_active":"2024-03-18T14:41:35.547332+0000","last_became_peered":"2024-03-18T14:41:35.547332+0000","last_unstale":"2024-03-18T14:41:40.559445+0000","last_undegraded":"2024-03-18T14:41:40.559445+0000","last_fullsized":"2024-03-18T14:41:40.559445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779000+0000","last_change":"2024-03-18T14:41:35.550445+0000","last_active":"2024-03-18T14:41:40.779000+0000","last_peered":"2024-03-18T14:41:40.779000+0000","last_clean":"2024-03-18T14:41:40.779000+0000","last_became_active":"2024-03-18T14:41:35.550120+0000","last_became_peered":"2024-03-18T14:41:35.550120+0000","last_unstale":"2024-03-18T14:41:40.779000+0000","last_undegraded":"2024-03-18T14:41:40.779000+0000","last_fullsized":"2024-03-18T14:41:40.779000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778964+0000","last_change":"2024-03-18T14:41:35.548653+0000","last_active":"2024-03-18T14:41:40.778964+0000","last_peered":"2024-03-18T14:41:40.778964+0000","last_clean":"2024-03-18T14:41:40.778964+0000","last_became_active":"2024-03-18T14:41:35.548305+0000","last_became_peered":"2024-03-18T14:41:35.548305+0000","last_unstale":"2024-03-18T14:41:40.778964+0000","last_undegraded":"2024-03-18T14:41:40.778964+0000","last_fullsized":"2024-03-18T14:41:40.778964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778662+0000","last_change":"2024-03-18T14:41:35.560778+0000","last_active":"2024-03-18T14:41:40.778662+0000","last_peered":"2024-03-18T14:41:40.778662+0000","last_clean":"2024-03-18T14:41:40.778662+0000","last_became_active":"2024-03-18T14:41:35.560478+0000","last_became_peered":"2024-03-18T14:41:35.560478+0000","last_unstale":"2024-03-18T14:41:40.778662+0000","last_undegraded":"2024-03-18T14:41:40.778662+0000","last_fullsized":"2024-03-18T14:41:40.778662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:41:58.961 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T14:41:58.961 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T14:41:58.962 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-18T14:41:58.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-18T14:41:58.962 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-18T14:41:59.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.097+0000 7f928e2d1640 1 Processor -- start 2024-03-18T14:41:59.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.097+0000 7f928e2d1640 1 -- start start 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f928e2d1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f928806cce0 0x7f928806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f928e2d1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f92880d1270 0x7f92880b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f928e2d1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92880cfd80 con 0x7f92880b17b0 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f928e2d1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f92880b2df0 con 0x7f928806cce0 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f928e2d1640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f928806d600 con 0x7f92880d1270 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f9287fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f928806cce0 0x7f928806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f92877fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f92880b17b0 0x7f92880ce3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53040/0 (socket says 172.21.15.6:53040) 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f9287fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f928806cce0 0x7f928806d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56336/0 (socket says 172.21.15.6:56336) 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f92877fe640 1 -- 172.21.15.6:0/2752924082 learned_addr learned my addr 172.21.15.6:0/2752924082 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f928c86e640 1 --2- 172.21.15.6:0/2752924082 >> v2:172.21.15.92:3300/0 conn(0x7f92880d1270 0x7f92880b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.098+0000 7f9287fff640 1 -- 172.21.15.6:0/2752924082 >> v2:172.21.15.92:3300/0 conn(0x7f92880d1270 msgr2=0x7f92880b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9287fff640 1 --2- 172.21.15.6:0/2752924082 >> v2:172.21.15.92:3300/0 conn(0x7f92880d1270 0x7f92880b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9287fff640 1 -- 172.21.15.6:0/2752924082 >> v1:172.21.15.6:6789/0 conn(0x7f92880b17b0 legacy=0x7f92880ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9287fff640 1 -- 172.21.15.6:0/2752924082 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92880b2270 con 0x7f928806cce0 2024-03-18T14:41:59.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9286ffd640 1 -- 172.21.15.6:0/2752924082 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415824247 0 0) 0x7f92880cfd80 con 0x7f92880b17b0 2024-03-18T14:41:59.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f928c86e640 1 --2- 172.21.15.6:0/2752924082 >> v2:172.21.15.92:3300/0 conn(0x7f92880d1270 0x7f92880b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:41:59.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9287fff640 1 --2- 172.21.15.6:0/2752924082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f928806d0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f928806e680 tx=0x7f927802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdc2c890071bc2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9286ffd640 1 -- 172.21.15.6:0/2752924082 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92780315a0 con 0x7f928806cce0 2024-03-18T14:41:59.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9286ffd640 1 -- 172.21.15.6:0/2752924082 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9278037070 con 0x7f928806cce0 2024-03-18T14:41:59.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.099+0000 7f9286ffd640 1 -- 172.21.15.6:0/2752924082 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278031960 con 0x7f928806cce0 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 -- 172.21.15.6:0/2752924082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 msgr2=0x7f928806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 --2- 172.21.15.6:0/2752924082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f928806d0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f928806e680 tx=0x7f927802b840 comp rx=0 tx=0).stop 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 -- 172.21.15.6:0/2752924082 shutdown_connections 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 --2- 172.21.15.6:0/2752924082 >> v2:172.21.15.92:3300/0 conn(0x7f92880d1270 0x7f92880b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 --2- 172.21.15.6:0/2752924082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f928806d0c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 -- 172.21.15.6:0/2752924082 >> 172.21.15.6:0/2752924082 conn(0x7f928808b130 msgr2=0x7f928808b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 -- 172.21.15.6:0/2752924082 shutdown_connections 2024-03-18T14:41:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.100+0000 7f928e2d1640 1 -- 172.21.15.6:0/2752924082 wait complete. 2024-03-18T14:41:59.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.101+0000 7f928e2d1640 1 Processor -- start 2024-03-18T14:41:59.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.101+0000 7f928e2d1640 1 -- start start 2024-03-18T14:41:59.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f928e2d1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f92881f8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f928e2d1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92880b17b0 0x7f928815fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f928e2d1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92880d1270 0x7f92880c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f928e2d1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f92880b2b10 con 0x7f928806cce0 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f928e2d1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92880b2830 con 0x7f92880b17b0 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f928e2d1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f928815e330 con 0x7f92880d1270 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f92881f8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f92881f8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56338/0 (socket says 172.21.15.6:56338) 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 -- 172.21.15.6:0/3583804072 learned_addr learned my addr 172.21.15.6:0/3583804072 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f92877fe640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92880b17b0 0x7f928815fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f928c86e640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92880d1270 0x7f92880c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 -- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92880d1270 msgr2=0x7f92880c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92880d1270 0x7f92880c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 -- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92880b17b0 msgr2=0x7f928815fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92880b17b0 0x7f928815fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.102+0000 7f9287fff640 1 -- 172.21.15.6:0/3583804072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928815f180 con 0x7f928806cce0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.103+0000 7f9287fff640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f92881f8080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9278009890 tx=0x7f92780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.103+0000 7f9284ff9640 1 -- 172.21.15.6:0/3583804072 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278032630 con 0x7f928806cce0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.103+0000 7f9284ff9640 1 -- 172.21.15.6:0/3583804072 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92780327f0 con 0x7f928806cce0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.103+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92880c8110 con 0x7f928806cce0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.103+0000 7f9284ff9640 1 -- 172.21.15.6:0/3583804072 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278032be0 con 0x7f928806cce0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.103+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f928815f400 con 0x7f928806cce0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.104+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f928809fa80 con 0x7f928806cce0 2024-03-18T14:41:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.104+0000 7f9284ff9640 1 -- 172.21.15.6:0/3583804072 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f927804c020 con 0x7f928806cce0 2024-03-18T14:41:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.105+0000 7f9284ff9640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f926806bc00 0x7f926806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.105+0000 7f9284ff9640 1 -- 172.21.15.6:0/3583804072 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f927802f020 con 0x7f928806cce0 2024-03-18T14:41:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.105+0000 7f92877fe640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f926806bc00 0x7f926806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.105+0000 7f92877fe640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f926806bc00 0x7f926806e0c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f928806ff90 tx=0x7f9270007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.109+0000 7f9284ff9640 1 -- 172.21.15.6:0/3583804072 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f928815f180 con 0x7f928806cce0 2024-03-18T14:41:59.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.340+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f928804f620 con 0x7f928806cce0 2024-03-18T14:41:59.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.343+0000 7f9284ff9640 1 -- 172.21.15.6:0/3583804072 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172578 (secure 0 0 0) 0x7f9278040070 con 0x7f928806cce0 2024-03-18T14:41:59.347 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:59.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f926806bc00 msgr2=0x7f926806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f926806bc00 0x7f926806e0c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f928806ff90 tx=0x7f9270007de0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 msgr2=0x7f92881f8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f92881f8080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9278009890 tx=0x7f92780046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 shutdown_connections 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f926806bc00 0x7f926806e0c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92880d1270 0x7f92880c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92880b17b0 0x7f928815fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 --2- 172.21.15.6:0/3583804072 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f928806cce0 0x7f92881f8080 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 >> 172.21.15.6:0/3583804072 conn(0x7f928808b130 msgr2=0x7f928815bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 shutdown_connections 2024-03-18T14:41:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.346+0000 7f928e2d1640 1 -- 172.21.15.6:0/3583804072 wait complete. 2024-03-18T14:41:59.358 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":4016547990},{"type":"v1","addr":"172.21.15.92:6833","nonce":4016547990}]},"active_addr":"172.21.15.92:6833/4016547990","active_change":"2024-03-18T14:41:15.370325+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":3717960838}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":974397033}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":1139350585}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":3757336072}]}]} 2024-03-18T14:41:59.361 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-18T14:41:59.361 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-18T14:41:59.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:59.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.461+0000 7f5badcb2640 1 Processor -- start 2024-03-18T14:41:59.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.461+0000 7f5badcb2640 1 -- start start 2024-03-18T14:41:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5badcb2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5ba815e170 0x7f5ba80b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5badcb2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ba80cef50 con 0x7f5ba80d0d90 2024-03-18T14:41:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5badcb2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ba80cf230 con 0x7f5ba80d3330 2024-03-18T14:41:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5badcb2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f5ba80b9b90 con 0x7f5ba815e170 2024-03-18T14:41:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ba80d0d90 0x7f5ba80b6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53050/0 (socket says 172.21.15.6:53050) 2024-03-18T14:41:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba7fff640 1 -- 172.21.15.6:0/805898955 learned_addr learned my addr 172.21.15.6:0/805898955 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:41:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba77fe640 1 --2- 172.21.15.6:0/805898955 >> v2:172.21.15.92:3300/0 conn(0x7f5ba815e170 0x7f5ba80b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba67fc640 1 -- 172.21.15.6:0/805898955 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2820649087 0 0) 0x7f5ba80cef50 con 0x7f5ba80d0d90 2024-03-18T14:41:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba67fc640 1 -- 172.21.15.6:0/805898955 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b90003520 con 0x7f5ba80d0d90 2024-03-18T14:41:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba67fc640 1 -- 172.21.15.6:0/805898955 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2638152746 0 0) 0x7f5ba80cf230 con 0x7f5ba80d3330 2024-03-18T14:41:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba67fc640 1 -- 172.21.15.6:0/805898955 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ba80cef50 con 0x7f5ba80d3330 2024-03-18T14:41:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba77fe640 1 -- 172.21.15.6:0/805898955 >> v1:172.21.15.92:6789/0 conn(0x7f5ba80d3330 legacy=0x7f5ba80d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba77fe640 1 -- 172.21.15.6:0/805898955 >> v1:172.21.15.6:6789/0 conn(0x7f5ba80d0d90 legacy=0x7f5ba80b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.462+0000 7f5ba77fe640 1 -- 172.21.15.6:0/805898955 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba80d2b30 con 0x7f5ba815e170 2024-03-18T14:41:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.463+0000 7f5ba77fe640 1 --2- 172.21.15.6:0/805898955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 0x7f5ba80b9650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b980097a0 tx=0x7f5b9802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85508396bc9e614b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.463+0000 7f5ba67fc640 1 -- 172.21.15.6:0/805898955 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98034070 con 0x7f5ba815e170 2024-03-18T14:41:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.463+0000 7f5ba67fc640 1 -- 172.21.15.6:0/805898955 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b98030610 con 0x7f5ba815e170 2024-03-18T14:41:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.463+0000 7f5ba67fc640 1 -- 172.21.15.6:0/805898955 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b980309c0 con 0x7f5ba815e170 2024-03-18T14:41:59.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.464+0000 7f5badcb2640 1 -- 172.21.15.6:0/805898955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 msgr2=0x7f5ba80b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.464+0000 7f5badcb2640 1 --2- 172.21.15.6:0/805898955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 0x7f5ba80b9650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b980097a0 tx=0x7f5b9802b960 comp rx=0 tx=0).stop 2024-03-18T14:41:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.464+0000 7f5badcb2640 1 -- 172.21.15.6:0/805898955 shutdown_connections 2024-03-18T14:41:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.464+0000 7f5badcb2640 1 --2- 172.21.15.6:0/805898955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 0x7f5ba80b9650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.464+0000 7f5badcb2640 1 -- 172.21.15.6:0/805898955 >> 172.21.15.6:0/805898955 conn(0x7f5ba806c900 msgr2=0x7f5ba806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.464+0000 7f5badcb2640 1 -- 172.21.15.6:0/805898955 shutdown_connections 2024-03-18T14:41:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.464+0000 7f5badcb2640 1 -- 172.21.15.6:0/805898955 wait complete. 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.465+0000 7f5badcb2640 1 Processor -- start 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.465+0000 7f5badcb2640 1 -- start start 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5badcb2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ba80d0d90 0x7f5ba80b9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5badcb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ba80d3330 0x7f5ba8160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5badcb2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 0x7f5ba80b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5badcb2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba80ceb70 con 0x7f5ba80d0d90 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5badcb2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba80d2820 con 0x7f5ba815e170 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5badcb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba80cecf0 con 0x7f5ba80d3330 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ba80d0d90 0x7f5ba80b9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ba80d0d90 0x7f5ba80b9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56352/0 (socket says 172.21.15.6:56352) 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 -- 172.21.15.6:0/3707730161 learned_addr learned my addr 172.21.15.6:0/3707730161 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba6ffd640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ba80d3330 0x7f5ba8160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba7fff640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 0x7f5ba80b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 -- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ba80d3330 msgr2=0x7f5ba8160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ba80d3330 0x7f5ba8160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 -- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 msgr2=0x7f5ba80b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 0x7f5ba80b7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.466+0000 7f5ba77fe640 1 -- 172.21.15.6:0/3707730161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba8161470 con 0x7f5ba80d0d90 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.467+0000 7f5ba77fe640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ba80d0d90 0x7f5ba80b9300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5b98000c00 tx=0x7f5b980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.467+0000 7f5b8ffff640 1 -- 172.21.15.6:0/3707730161 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98034040 con 0x7f5ba80d0d90 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.467+0000 7f5b8ffff640 1 -- 172.21.15.6:0/3707730161 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b98030410 con 0x7f5ba80d0d90 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.467+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba815f590 con 0x7f5ba80d0d90 2024-03-18T14:41:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.467+0000 7f5b8ffff640 1 -- 172.21.15.6:0/3707730161 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98006e10 con 0x7f5ba80d0d90 2024-03-18T14:41:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.467+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba80cbae0 con 0x7f5ba80d0d90 2024-03-18T14:41:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.468+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba808afe0 con 0x7f5ba80d0d90 2024-03-18T14:41:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.468+0000 7f5b8ffff640 1 -- 172.21.15.6:0/3707730161 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5b98037070 con 0x7f5ba80d0d90 2024-03-18T14:41:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.469+0000 7f5b8ffff640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5b9406bbd0 0x7f5b9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.469+0000 7f5b8ffff640 1 -- 172.21.15.6:0/3707730161 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5b980aca80 con 0x7f5ba80d0d90 2024-03-18T14:41:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.469+0000 7f5ba6ffd640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5b9406bbd0 0x7f5b9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.469+0000 7f5ba6ffd640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5b9406bbd0 0x7f5b9406e090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c007fd0 tx=0x7f5b9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.473+0000 7f5b8ffff640 1 -- 172.21.15.6:0/3707730161 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5b9807ac70 con 0x7f5ba80d0d90 2024-03-18T14:41:59.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.663+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ba815cc30 con 0x7f5ba80d0d90 2024-03-18T14:41:59.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.664+0000 7f5b8ffff640 1 -- 172.21.15.6:0/3707730161 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f5b9807ed10 con 0x7f5ba80d0d90 2024-03-18T14:41:59.666 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:59.666 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:59.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5b9406bbd0 msgr2=0x7f5b9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5b9406bbd0 0x7f5b9406e090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c007fd0 tx=0x7f5b9c009040 comp rx=0 tx=0).stop 2024-03-18T14:41:59.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ba80d0d90 msgr2=0x7f5ba80b9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ba80d0d90 0x7f5ba80b9300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5b98000c00 tx=0x7f5b980046c0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 shutdown_connections 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5b9406bbd0 0x7f5b9406e090 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba815e170 0x7f5ba80b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ba80d3330 0x7f5ba8160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 --2- 172.21.15.6:0/3707730161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ba80d0d90 0x7f5ba80b9300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.666+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 >> 172.21.15.6:0/3707730161 conn(0x7f5ba806c900 msgr2=0x7f5ba80c81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.667+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 shutdown_connections 2024-03-18T14:41:59.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.667+0000 7f5badcb2640 1 -- 172.21.15.6:0/3707730161 wait complete. 2024-03-18T14:41:59.679 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-18T14:41:59.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:41:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.779+0000 7f5fb83d4640 1 Processor -- start 2024-03-18T14:41:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.779+0000 7f5fb83d4640 1 -- start start 2024-03-18T14:41:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb83d4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5fb00b2260 0x7f5fb00cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb83d4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb00b28b0 con 0x7f5fb015def0 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb83d4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb00b2b90 con 0x7f5fb006d3f0 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb83d4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb00d0540 con 0x7f5fb00b2260 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb6971640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5fb00b2260 0x7f5fb00cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb6971640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5fb00b2260 0x7f5fb00cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56360/0 (socket says 172.21.15.6:56360) 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb6971640 1 -- 172.21.15.6:0/1400602711 learned_addr learned my addr 172.21.15.6:0/1400602711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.780+0000 7f5fb6170640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5fb015def0 0x7f5fb006cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53058/0 (socket says 172.21.15.6:53058) 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.781+0000 7f5fb6971640 1 -- 172.21.15.6:0/1400602711 >> v1:172.21.15.92:6789/0 conn(0x7f5fb006d3f0 legacy=0x7f5fb00b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.781+0000 7f5fb6971640 1 -- 172.21.15.6:0/1400602711 >> v1:172.21.15.6:6789/0 conn(0x7f5fb015def0 legacy=0x7f5fb006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.781+0000 7f5fb516e640 1 -- 172.21.15.6:0/1400602711 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783374180 0 0) 0x7f5fb00b28b0 con 0x7f5fb015def0 2024-03-18T14:41:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.781+0000 7f5fb6971640 1 -- 172.21.15.6:0/1400602711 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb00d01d0 con 0x7f5fb00b2260 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.781+0000 7f5fb6971640 1 --2- 172.21.15.6:0/1400602711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb00b2260 0x7f5fb00cc1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00d06c0 tx=0x7f5fa802bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ff24fc29aef651 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb516e640 1 -- 172.21.15.6:0/1400602711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa802cb80 con 0x7f5fb00b2260 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb516e640 1 -- 172.21.15.6:0/1400602711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa802cd20 con 0x7f5fb00b2260 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb516e640 1 -- 172.21.15.6:0/1400602711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8034610 con 0x7f5fb00b2260 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb83d4640 1 -- 172.21.15.6:0/1400602711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb00b2260 msgr2=0x7f5fb00cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/1400602711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb00b2260 0x7f5fb00cc1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00d06c0 tx=0x7f5fa802bfe0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb83d4640 1 -- 172.21.15.6:0/1400602711 shutdown_connections 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/1400602711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb00b2260 0x7f5fb00cc1a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb83d4640 1 -- 172.21.15.6:0/1400602711 >> 172.21.15.6:0/1400602711 conn(0x7f5fb009ae20 msgr2=0x7f5fb008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb83d4640 1 -- 172.21.15.6:0/1400602711 shutdown_connections 2024-03-18T14:41:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.782+0000 7f5fb83d4640 1 -- 172.21.15.6:0/1400602711 wait complete. 2024-03-18T14:41:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.783+0000 7f5fb83d4640 1 Processor -- start 2024-03-18T14:41:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.783+0000 7f5fb83d4640 1 -- start start 2024-03-18T14:41:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb83d4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb006d3f0 0x7f5fb015efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb83d4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb00b2260 0x7f5fb00d1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb83d4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb015def0 0x7f5fb00cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb83d4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb00d09d0 con 0x7f5fb015def0 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb83d4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb015e2d0 con 0x7f5fb00b2260 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb83d4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb00d0b50 con 0x7f5fb006d3f0 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb6971640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb015def0 0x7f5fb00cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb6971640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb015def0 0x7f5fb00cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56368/0 (socket says 172.21.15.6:56368) 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb6971640 1 -- 172.21.15.6:0/956867270 learned_addr learned my addr 172.21.15.6:0/956867270 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb6170640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb006d3f0 0x7f5fb015efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.784+0000 7f5fb596f640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb00b2260 0x7f5fb00d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb6971640 1 -- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb006d3f0 msgr2=0x7f5fb015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb6971640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb006d3f0 0x7f5fb015efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb6971640 1 -- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb00b2260 msgr2=0x7f5fb00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb6971640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb00b2260 0x7f5fb00d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb6971640 1 -- 172.21.15.6:0/956867270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb00b63b0 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb6971640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb015def0 0x7f5fb00cb370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009f40 tx=0x7f5fa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fa6ffd640 1 -- 172.21.15.6:0/956867270 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8043020 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fa6ffd640 1 -- 172.21.15.6:0/956867270 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa803e450 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb00b66f0 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fa6ffd640 1 -- 172.21.15.6:0/956867270 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8009510 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.785+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb00b4720 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.786+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb009fa90 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.787+0000 7f5fa6ffd640 1 -- 172.21.15.6:0/956867270 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5fa8046020 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.787+0000 7f5fa6ffd640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5f9406bbd0 0x7f5f9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.787+0000 7f5fa6ffd640 1 -- 172.21.15.6:0/956867270 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5fa80ad570 con 0x7f5fb015def0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.787+0000 7f5fb6170640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5f9406bbd0 0x7f5f9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:41:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.788+0000 7f5fb6170640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5f9406bbd0 0x7f5f9406e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5fac000f60 tx=0x7f5fac007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:41:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.791+0000 7f5fa6ffd640 1 -- 172.21.15.6:0/956867270 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5fa804d030 con 0x7f5fb015def0 2024-03-18T14:41:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.973+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fb004f620 con 0x7f5fb015def0 2024-03-18T14:41:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.974+0000 7f5fa6ffd640 1 -- 172.21.15.6:0/956867270 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f5fb00b66f0 con 0x7f5fb015def0 2024-03-18T14:41:59.976 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:41:59.976 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:41:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5f9406bbd0 msgr2=0x7f5f9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5f9406bbd0 0x7f5f9406e090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5fac000f60 tx=0x7f5fac007e40 comp rx=0 tx=0).stop 2024-03-18T14:41:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb015def0 msgr2=0x7f5fb00cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:41:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb015def0 0x7f5fb00cb370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009f40 tx=0x7f5fa8004a40 comp rx=0 tx=0).stop 2024-03-18T14:41:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 shutdown_connections 2024-03-18T14:41:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5f9406bbd0 0x7f5f9406e090 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fb015def0 0x7f5fb00cb370 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb00b2260 0x7f5fb00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 --2- 172.21.15.6:0/956867270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb006d3f0 0x7f5fb015efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:41:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 >> 172.21.15.6:0/956867270 conn(0x7f5fb009ae20 msgr2=0x7f5fb00ce3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:41:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 shutdown_connections 2024-03-18T14:41:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:41:59.976+0000 7f5fb83d4640 1 -- 172.21.15.6:0/956867270 wait complete. 2024-03-18T14:41:59.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-18T14:41:59.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-18T14:41:59.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-18T14:41:59.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-18T14:41:59.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-18T14:41:59.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-18T14:41:59.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-18T14:41:59.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-18T14:42:00.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7fb979e64640 1 Processor -- start 2024-03-18T14:42:00.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7fec803a9640 1 Processor -- start 2024-03-18T14:42:00.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7f614299f640 1 Processor -- start 2024-03-18T14:42:00.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7fb979e64640 1 -- start start 2024-03-18T14:42:00.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7fec803a9640 1 -- start start 2024-03-18T14:42:00.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7fdacb59e640 1 Processor -- start 2024-03-18T14:42:00.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7f7f1e085640 1 Processor -- start 2024-03-18T14:42:00.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7f7f1e085640 1 -- start start 2024-03-18T14:42:00.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7fdacb59e640 1 -- start start 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7fbd07268640 1 Processor -- start 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fbd07268640 1 -- start start 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd07268640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbd0015c830 0x7fbd00163120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd07268640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbd000b39e0 0x7fbd0006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd07268640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd000b2e30 con 0x7fbd000b2870 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd07268640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbd000af240 con 0x7fbd0015c830 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd07268640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbd0015cc10 con 0x7fbd000b39e0 2024-03-18T14:42:00.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fec803a9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fec7806e0b0 0x7fec780b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fec803a9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec7806f650 0x7fec780ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fec803a9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7806e5b0 con 0x7fec78072050 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fec803a9640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fec780b4790 con 0x7fec7806e0b0 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fec803a9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fec7806f1c0 con 0x7fec7806f650 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fec7e145640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fec78072050 0x7fec78072430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45724/0 (socket says 172.21.15.6:45724) 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fec7e145640 1 -- 172.21.15.6:0/3179035341 learned_addr learned my addr 172.21.15.6:0/3179035341 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fec7e946640 1 --2- 172.21.15.6:0/3179035341 >> v2:172.21.15.92:3300/0 conn(0x7fec7806f650 0x7fec780ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3913467517 0 0) 0x7fec7806e5b0 con 0x7fec78072050 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec64003520 con 0x7fec78072050 2024-03-18T14:42:00.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.165+0000 7f614299f640 1 -- start start 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f614299f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61340df800 0x7f6134050740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f614119c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61340df800 0x7f6134050740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f614119c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61340df800 0x7f6134050740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56390/0 (socket says 172.21.15.6:56390) 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f7f1e085640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7f180ce740 0x7f7f180d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f7f1e085640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7f180d0c20 0x7f7f180d1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f7f1e085640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f181630f0 con 0x7f7f18162850 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f7f1e085640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7f180d1590 con 0x7f7f180d0c20 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f7f1e085640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7f181633d0 con 0x7f7f180ce740 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f7f17fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7f180d0c20 0x7f7f180d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f7f17fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7f180d0c20 0x7f7f180d1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56402/0 (socket says 172.21.15.6:56402) 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f7f17fff640 1 -- 172.21.15.6:0/2460017368 learned_addr learned my addr 172.21.15.6:0/2460017368 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fb979e64640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c0530a0 0x7fb96c0e0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fb979e64640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb96c0538d0 con 0x7fb96c00aad0 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fb979e64640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb96c053bb0 con 0x7fb96c04fa70 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fb979e64640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb96c0e5ff0 con 0x7fb96c0530a0 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fb973fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c0530a0 0x7fb96c0e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fb973fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c0530a0 0x7fb96c0e0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50612/0 (socket says 172.21.15.6:50612) 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fb973fff640 1 -- 172.21.15.6:0/1468620033 learned_addr learned my addr 172.21.15.6:0/1468620033 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f3bc3fff640 1 Processor -- start 2024-03-18T14:42:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fa6e507e640 1 Processor -- start 2024-03-18T14:42:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7fa6e507e640 1 -- start start 2024-03-18T14:42:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdacb59e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdac4006a30 0x7fdac4006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdacb59e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac4053720 con 0x7fdac40ead00 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdacb59e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac4053a00 con 0x7fdac4005ee0 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdacb59e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fdac4054330 con 0x7fdac4006a30 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdacad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdac4006a30 0x7fdac4006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdacad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdac4006a30 0x7fdac4006ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56420/0 (socket says 172.21.15.6:56420) 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdacad9d640 1 -- 172.21.15.6:0/754053831 learned_addr learned my addr 172.21.15.6:0/754053831 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fdac9d9b640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdac4005ee0 0x7fdac4006320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45758/0 (socket says 172.21.15.6:45758) 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fb973fff640 1 -- 172.21.15.6:0/1468620033 >> v1:172.21.15.92:6789/0 conn(0x7fb96c04fa70 legacy=0x7fb96c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fb973fff640 1 -- 172.21.15.6:0/1468620033 >> v1:172.21.15.6:6789/0 conn(0x7fb96c00aad0 legacy=0x7fb96c0dee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fb973fff640 1 -- 172.21.15.6:0/1468620033 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c053e90 con 0x7fb96c0530a0 2024-03-18T14:42:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f7f177fe640 1 --2- 172.21.15.6:0/2460017368 >> v2:172.21.15.92:3300/0 conn(0x7f7f180ce740 0x7f7f180d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd05805640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbd000b39e0 0x7fbd0006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd05805640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbd000b39e0 0x7fbd0006e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:39994/0 (socket says 172.21.15.6:39994) 2024-03-18T14:42:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd05805640 1 -- 172.21.15.6:0/2887593815 learned_addr learned my addr 172.21.15.6:0/2887593815 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fbd05004640 1 --2- 172.21.15.6:0/2887593815 >> v2:172.21.15.6:3300/0 conn(0x7fbd0015c830 0x7fbd00163120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.166+0000 7f3bc3fff640 1 -- start start 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f3bc3fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc0df3e0 0x7f3bbc04fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f3bc3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3bbc050330 0x7f3bbc0530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f3bc3fff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bbc054d40 con 0x7f3bbc0df000 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f3bc3fff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3bbc054640 con 0x7f3bbc050330 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7f3bc3fff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3bbc006de0 con 0x7f3bbc0df3e0 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f7f17fff640 1 -- 172.21.15.6:0/2460017368 >> v2:172.21.15.92:3300/0 conn(0x7f7f180ce740 msgr2=0x7f7f180d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f7f17fff640 1 --2- 172.21.15.6:0/2460017368 >> v2:172.21.15.92:3300/0 conn(0x7f7f180ce740 0x7f7f180d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f7f17fff640 1 -- 172.21.15.6:0/2460017368 >> v1:172.21.15.6:6789/0 conn(0x7f7f18162850 legacy=0x7f7f18166640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f7f17fff640 1 -- 172.21.15.6:0/2460017368 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f181636b0 con 0x7f7f180d0c20 2024-03-18T14:42:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fa6e507e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa6d80544a0 0x7fa6d803c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc27fc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc0df3e0 0x7f3bbc04fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc27fc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc0df3e0 0x7f3bbc04fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50628/0 (socket says 172.21.15.6:50628) 2024-03-18T14:42:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc27fc640 1 -- 172.21.15.6:0/3076950823 learned_addr learned my addr 172.21.15.6:0/3076950823 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fa6e507e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d8058980 con 0x7fa6d80e3040 2024-03-18T14:42:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fbd05805640 1 -- 172.21.15.6:0/2887593815 >> v2:172.21.15.6:3300/0 conn(0x7fbd0015c830 msgr2=0x7fbd00163120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fbd05805640 1 --2- 172.21.15.6:0/2887593815 >> v2:172.21.15.6:3300/0 conn(0x7fbd0015c830 0x7fbd00163120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fbd05805640 1 -- 172.21.15.6:0/2887593815 >> v1:172.21.15.92:6789/0 conn(0x7fbd000b2870 legacy=0x7fbd000b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fbd05805640 1 -- 172.21.15.6:0/2887593815 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd00165ce0 con 0x7fbd000b39e0 2024-03-18T14:42:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc37fe640 1 --2- 172.21.15.6:0/3076950823 >> v2:172.21.15.6:3300/0 conn(0x7f3bbc050330 0x7f3bbc0530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2633734856 0 0) 0x7fec64003520 con 0x7fec78072050 2024-03-18T14:42:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fa6e507e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d8058c60 con 0x7fa6d80e3420 2024-03-18T14:42:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.167+0000 7fa6e507e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa6d8006860 con 0x7fa6d80544a0 2024-03-18T14:42:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fa6ded9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6d80e3040 0x7fa6d8058130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53108/0 (socket says 172.21.15.6:53108) 2024-03-18T14:42:00.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 >> v2:172.21.15.92:3300/0 conn(0x7fec7806f650 msgr2=0x7fec780ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 --2- 172.21.15.6:0/3179035341 >> v2:172.21.15.92:3300/0 conn(0x7fec7806f650 0x7fec780ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 >> v2:172.21.15.6:3300/0 conn(0x7fec7806e0b0 msgr2=0x7fec780b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 --2- 172.21.15.6:0/3179035341 >> v2:172.21.15.6:3300/0 conn(0x7fec7806e0b0 0x7fec780b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7806e890 con 0x7fec78072050 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fec700019f0 con 0x7fec78072050 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 >> v1:172.21.15.92:6789/0 conn(0x7fec78072050 legacy=0x7fec78072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec640039b0 0x7fec64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec64004800 0x7fec64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec640051c0 0x7fec6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fec780b4790 con 0x7fec640051c0 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fec7806f1c0 con 0x7fec64004800 2024-03-18T14:42:00.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fec6400c190 con 0x7fec640039b0 2024-03-18T14:42:00.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4033613096 0 0) 0x7fdac4053a00 con 0x7fdac4005ee0 2024-03-18T14:42:00.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdab4003520 con 0x7fdac4005ee0 2024-03-18T14:42:00.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fec7d944640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec64004800 0x7fec64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fec7e145640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec640039b0 0x7fec64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc27fc640 1 -- 172.21.15.6:0/3076950823 >> v2:172.21.15.6:3300/0 conn(0x7f3bbc050330 msgr2=0x7f3bbc0530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc27fc640 1 --2- 172.21.15.6:0/3076950823 >> v2:172.21.15.6:3300/0 conn(0x7f3bbc050330 0x7f3bbc0530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc27fc640 1 -- 172.21.15.6:0/3076950823 >> v1:172.21.15.6:6789/0 conn(0x7f3bbc0df000 legacy=0x7f3bbc04f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7f3bc27fc640 1 -- 172.21.15.6:0/3076950823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bbc050770 con 0x7f3bbc0df3e0 2024-03-18T14:42:00.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fa6ded9d640 1 -- 172.21.15.6:0/1000312795 learned_addr learned my addr 172.21.15.6:0/1000312795 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:00.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.168+0000 7fa6df59e640 1 --2- 172.21.15.6:0/1000312795 >> v2:172.21.15.6:3300/0 conn(0x7fa6d80544a0 0x7fa6d803c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3506501551 0 0) 0x7fdab4003520 con 0x7fdac4005ee0 2024-03-18T14:42:00.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614299f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61340526a0 con 0x7f6134038520 2024-03-18T14:42:00.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614299f640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6134053850 con 0x7f6134054200 2024-03-18T14:42:00.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614299f640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6134039080 con 0x7f61340df800 2024-03-18T14:42:00.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614199d640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6134054200 0x7f6134053140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45744/0 (socket says 172.21.15.6:45744) 2024-03-18T14:42:00.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614199d640 1 -- 172.21.15.6:0/2205226904 learned_addr learned my addr 172.21.15.6:0/2205226904 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:00.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 >> v2:172.21.15.6:3300/0 conn(0x7fdac4006a30 msgr2=0x7fdac4006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fec7d944640 1 -- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec640039b0 msgr2=0x7fec64003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fec7d944640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec640039b0 0x7fec64003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fec7d944640 1 -- 172.21.15.6:0/3179035341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec640051c0 msgr2=0x7fec6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fa6df59e640 1 -- 172.21.15.6:0/1000312795 >> v1:172.21.15.92:6789/0 conn(0x7fa6d80e3420 legacy=0x7fa6d8053d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T14:42:00.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fec7d944640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec640051c0 0x7fec6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fa6df59e640 1 -- 172.21.15.6:0/1000312795 >> v1:172.21.15.6:6789/0 conn(0x7fa6d80e3040 legacy=0x7fa6d8058130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fa6df59e640 1 -- 172.21.15.6:0/1000312795 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d8006210 con 0x7fa6d80544a0 2024-03-18T14:42:00.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fb973fff640 1 --2- 172.21.15.6:0/1468620033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c0530a0 0x7fb96c0e0100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb968013180 tx=0x7fb9680341a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2975c9fac2683fbc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fec7d944640 1 -- 172.21.15.6:0/3179035341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7806e890 con 0x7fec64004800 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fbd05805640 1 --2- 172.21.15.6:0/2887593815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 0x7fbd0006e0b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd000b2d20 tx=0x7fbcfc02b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133cff8ec66861b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fec7e946640 1 -- 172.21.15.6:0/3179035341 reap_dead start 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 --2- 172.21.15.6:0/754053831 >> v2:172.21.15.6:3300/0 conn(0x7fdac4006a30 0x7fdac4006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 >> v1:172.21.15.6:6789/0 conn(0x7fdac40ead00 legacy=0x7fdac404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac4053ce0 con 0x7fdac4005ee0 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fdac0001670 con 0x7fdac4005ee0 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 >> v1:172.21.15.92:6789/0 conn(0x7fdac4005ee0 legacy=0x7fdac4006320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdab4003a00 0x7fdab4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fdac959a640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4004800 0x7fdab4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fb9737fe640 1 -- 172.21.15.6:0/1468620033 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9680110d0 con 0x7fb96c0530a0 2024-03-18T14:42:00.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7fb9737fe640 1 -- 172.21.15.6:0/1468620033 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb968011270 con 0x7fb96c0530a0 2024-03-18T14:42:00.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f7f17fff640 1 --2- 172.21.15.6:0/2460017368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f180d0c20 0x7f7f180d1050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7f180b5f30 tx=0x7f7f0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc730ad217ce4af7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdaca59c640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdab4003a00 0x7fdab4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f3bc27fc640 1 --2- 172.21.15.6:0/3076950823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc0df3e0 0x7f3bbc04fdf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8008870 tx=0x7f3bb802dee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d20e29fffa4f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdac9d9b640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4004800 0x7fdab4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdac959a640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdab40052e0 0x7fdab400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdac4054330 con 0x7fdab4004800 2024-03-18T14:42:00.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f3bc1ffb640 1 -- 172.21.15.6:0/3076950823 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8034070 con 0x7f3bbc0df3e0 2024-03-18T14:42:00.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f3bc1ffb640 1 -- 172.21.15.6:0/3076950823 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb802ec90 con 0x7f3bbc0df3e0 2024-03-18T14:42:00.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f3bc1ffb640 1 -- 172.21.15.6:0/3076950823 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8038620 con 0x7f3bbc0df3e0 2024-03-18T14:42:00.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdab400c230 con 0x7fdab4003a00 2024-03-18T14:42:00.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fb9737fe640 1 -- 172.21.15.6:0/1468620033 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968011640 con 0x7fb96c0530a0 2024-03-18T14:42:00.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fb979e64640 1 -- 172.21.15.6:0/1468620033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c0530a0 msgr2=0x7fb96c0e0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fb979e64640 1 --2- 172.21.15.6:0/1468620033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c0530a0 0x7fb96c0e0100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb968013180 tx=0x7fb9680341a0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdab400c430 con 0x7fdab40052e0 2024-03-18T14:42:00.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdacad9d640 1 --2- 172.21.15.6:0/754053831 >> v2:172.21.15.6:3300/0 conn(0x7fdac4006a30 0x7fdac4006ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:00.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fb979e64640 1 -- 172.21.15.6:0/1468620033 shutdown_connections 2024-03-18T14:42:00.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fb979e64640 1 --2- 172.21.15.6:0/1468620033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c0530a0 0x7fb96c0e0100 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fb979e64640 1 -- 172.21.15.6:0/1468620033 >> 172.21.15.6:0/1468620033 conn(0x7fb96c009fb0 msgr2=0x7fb96c00a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fa6df59e640 1 --2- 172.21.15.6:0/1000312795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 0x7fa6d803c180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80069e0 tx=0x7fa6d402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6765e4998770532d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614119c640 1 -- 172.21.15.6:0/2205226904 >> v1:172.21.15.92:6789/0 conn(0x7f6134054200 legacy=0x7f6134053140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614119c640 1 -- 172.21.15.6:0/2205226904 >> v1:172.21.15.6:6789/0 conn(0x7f6134038520 legacy=0x7f6134038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.169+0000 7f614119c640 1 -- 172.21.15.6:0/2205226904 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6134053c80 con 0x7f61340df800 2024-03-18T14:42:00.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdacad9d640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdab40052e0 0x7fdab400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdaca59c640 1 -- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdab40052e0 msgr2=0x7fdab400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdaca59c640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdab40052e0 0x7fdab400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fdaca59c640 1 -- 172.21.15.6:0/754053831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4004800 msgr2=0x7fdab4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f7f167fc640 1 -- 172.21.15.6:0/2460017368 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c035020 con 0x7f7f180d0c20 2024-03-18T14:42:00.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f7f167fc640 1 -- 172.21.15.6:0/2460017368 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0c031870 con 0x7f7f180d0c20 2024-03-18T14:42:00.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7f7f167fc640 1 -- 172.21.15.6:0/2460017368 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c031c40 con 0x7f7f180d0c20 2024-03-18T14:42:00.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fb979e64640 1 -- 172.21.15.6:0/1468620033 shutdown_connections 2024-03-18T14:42:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fbcf7fff640 1 -- 172.21.15.6:0/2887593815 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc034070 con 0x7fbd000b39e0 2024-03-18T14:42:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.170+0000 7fec7d944640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec64004800 0x7fec64004c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fec68009860 tx=0x7fec64005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f1e2b52eee56728 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fbcf7fff640 1 -- 172.21.15.6:0/2887593815 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcfc030880 con 0x7fbd000b39e0 2024-03-18T14:42:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fbcf7fff640 1 -- 172.21.15.6:0/2887593815 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc030c90 con 0x7fbd000b39e0 2024-03-18T14:42:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fdaca59c640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4004800 0x7fdab4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fdaca59c640 1 -- 172.21.15.6:0/754053831 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab4003520 con 0x7fdab4003a00 2024-03-18T14:42:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68037070 con 0x7fec64004800 2024-03-18T14:42:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec68032620 con 0x7fec64004800 2024-03-18T14:42:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fdacad9d640 1 -- 172.21.15.6:0/754053831 reap_dead start 2024-03-18T14:42:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fec7d143640 1 -- 172.21.15.6:0/3179035341 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68009460 con 0x7fec64004800 2024-03-18T14:42:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f614119c640 1 --2- 172.21.15.6:0/2205226904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61340df800 0x7f6134050740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f612c0097a0 tx=0x7f612c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bdb1cfe4d6a55b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fa6ddd9b640 1 -- 172.21.15.6:0/1000312795 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4031780 con 0x7fa6d80544a0 2024-03-18T14:42:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fa6ddd9b640 1 -- 172.21.15.6:0/1000312795 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d4037070 con 0x7fa6d80544a0 2024-03-18T14:42:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fa6ddd9b640 1 -- 172.21.15.6:0/1000312795 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4031b40 con 0x7fa6d80544a0 2024-03-18T14:42:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 -- 172.21.15.6:0/3076950823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc0df3e0 msgr2=0x7f3bbc04fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 --2- 172.21.15.6:0/3076950823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc0df3e0 0x7f3bbc04fdf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8008870 tx=0x7f3bb802dee0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 -- 172.21.15.6:0/3076950823 shutdown_connections 2024-03-18T14:42:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 --2- 172.21.15.6:0/3076950823 >> v2:172.21.15.6:3300/0 conn(0x7f3bbc050330 0x7f3bbc0530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 --2- 172.21.15.6:0/3076950823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc0df3e0 0x7f3bbc04fdf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 -- 172.21.15.6:0/3076950823 >> 172.21.15.6:0/3076950823 conn(0x7f3bbc00a400 msgr2=0x7f3bbc00a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f614099b640 1 -- 172.21.15.6:0/2205226904 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c035020 con 0x7f61340df800 2024-03-18T14:42:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 -- 172.21.15.6:0/3076950823 shutdown_connections 2024-03-18T14:42:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f614099b640 1 -- 172.21.15.6:0/2205226904 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f612c031830 con 0x7f61340df800 2024-03-18T14:42:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f614099b640 1 -- 172.21.15.6:0/2205226904 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c031c00 con 0x7f61340df800 2024-03-18T14:42:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f7f1e085640 1 -- 172.21.15.6:0/2460017368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f180d0c20 msgr2=0x7f7f180d1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f7f1e085640 1 --2- 172.21.15.6:0/2460017368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f180d0c20 0x7f7f180d1050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7f180b5f30 tx=0x7f7f0c02b910 comp rx=0 tx=0).stop 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fbd07268640 1 -- 172.21.15.6:0/2887593815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 msgr2=0x7fbd0006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fbd07268640 1 --2- 172.21.15.6:0/2887593815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 0x7fbd0006e0b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd000b2d20 tx=0x7fbcfc02b9c0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fbd07268640 1 -- 172.21.15.6:0/2887593815 shutdown_connections 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fbd07268640 1 --2- 172.21.15.6:0/2887593815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 0x7fbd0006e0b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7fbd07268640 1 --2- 172.21.15.6:0/2887593815 >> v2:172.21.15.6:3300/0 conn(0x7fbd0015c830 0x7fbd00163120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fbd07268640 1 -- 172.21.15.6:0/2887593815 >> 172.21.15.6:0/2887593815 conn(0x7fbd0006b1c0 msgr2=0x7fbd0006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f7f1e085640 1 -- 172.21.15.6:0/2460017368 shutdown_connections 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f7f1e085640 1 --2- 172.21.15.6:0/2460017368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f180d0c20 0x7f7f180d1050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f7f1e085640 1 --2- 172.21.15.6:0/2460017368 >> v2:172.21.15.92:3300/0 conn(0x7f7f180ce740 0x7f7f180d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f7f1e085640 1 -- 172.21.15.6:0/2460017368 >> 172.21.15.6:0/2460017368 conn(0x7f7f18072ce0 msgr2=0x7f7f180730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fb979e64640 1 -- 172.21.15.6:0/1468620033 wait complete. 2024-03-18T14:42:00.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fec803a9640 1 -- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec64004800 msgr2=0x7fec64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fec803a9640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec64004800 0x7fec64004c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fec68009860 tx=0x7fec64005830 comp rx=0 tx=0).stop 2024-03-18T14:42:00.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f7f1e085640 1 -- 172.21.15.6:0/2460017368 shutdown_connections 2024-03-18T14:42:00.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f7f1e085640 1 -- 172.21.15.6:0/2460017368 wait complete. 2024-03-18T14:42:00.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.171+0000 7f3bc3fff640 1 -- 172.21.15.6:0/3076950823 wait complete. 2024-03-18T14:42:00.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fa6e507e640 1 -- 172.21.15.6:0/1000312795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 msgr2=0x7fa6d803c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fec803a9640 1 -- 172.21.15.6:0/3179035341 shutdown_connections 2024-03-18T14:42:00.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fec803a9640 1 --2- 172.21.15.6:0/3179035341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec64004800 0x7fec64004c80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fec803a9640 1 -- 172.21.15.6:0/3179035341 >> 172.21.15.6:0/3179035341 conn(0x7fec780b1b90 msgr2=0x7fec780b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fa6e507e640 1 --2- 172.21.15.6:0/1000312795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 0x7fa6d803c180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80069e0 tx=0x7fa6d402b820 comp rx=0 tx=0).stop 2024-03-18T14:42:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fa6e507e640 1 -- 172.21.15.6:0/1000312795 shutdown_connections 2024-03-18T14:42:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fa6e507e640 1 --2- 172.21.15.6:0/1000312795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 0x7fa6d803c180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fa6e507e640 1 -- 172.21.15.6:0/1000312795 >> 172.21.15.6:0/1000312795 conn(0x7fa6d803aa40 msgr2=0x7fa6d803ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fec803a9640 1 -- 172.21.15.6:0/3179035341 shutdown_connections 2024-03-18T14:42:00.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fbd07268640 1 -- 172.21.15.6:0/2887593815 shutdown_connections 2024-03-18T14:42:00.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fbd07268640 1 -- 172.21.15.6:0/2887593815 wait complete. 2024-03-18T14:42:00.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fec803a9640 1 -- 172.21.15.6:0/3179035341 wait complete. 2024-03-18T14:42:00.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f614299f640 1 -- 172.21.15.6:0/2205226904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61340df800 msgr2=0x7f6134050740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f614299f640 1 --2- 172.21.15.6:0/2205226904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61340df800 0x7f6134050740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f612c0097a0 tx=0x7f612c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fbd07268640 1 Processor -- start 2024-03-18T14:42:00.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f614299f640 1 -- 172.21.15.6:0/2205226904 shutdown_connections 2024-03-18T14:42:00.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f614299f640 1 --2- 172.21.15.6:0/2205226904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61340df800 0x7f6134050740 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f612c0097a0 tx=0x7f612c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7f614299f640 1 -- 172.21.15.6:0/2205226904 >> 172.21.15.6:0/2205226904 conn(0x7f6134009fb0 msgr2=0x7f613400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7f7f1e085640 1 Processor -- start 2024-03-18T14:42:00.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fb979e64640 1 Processor -- start 2024-03-18T14:42:00.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fa6e507e640 1 -- 172.21.15.6:0/1000312795 shutdown_connections 2024-03-18T14:42:00.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fa6e507e640 1 -- 172.21.15.6:0/1000312795 wait complete. 2024-03-18T14:42:00.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7f614299f640 1 -- 172.21.15.6:0/2205226904 shutdown_connections 2024-03-18T14:42:00.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7fbd07268640 1 -- start start 2024-03-18T14:42:00.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7fec803a9640 1 Processor -- start 2024-03-18T14:42:00.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7f614299f640 1 -- 172.21.15.6:0/2205226904 wait complete. 2024-03-18T14:42:00.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7fa6e507e640 1 Processor -- start 2024-03-18T14:42:00.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fa6e507e640 1 -- start start 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fbd07268640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd000b2870 0x7fbd0014dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fbd07268640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 0x7fbd0015e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fbd07268640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd0015c830 0x7fbd0015ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fbd07268640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbd001662a0 con 0x7fbd0015c830 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fbd07268640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbd00163660 con 0x7fbd000b39e0 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fbd07268640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbd00166580 con 0x7fbd000b2870 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7f7f1e085640 1 -- start start 2024-03-18T14:42:00.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f7f1e085640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f180ce740 0x7f7f18158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7fb979e64640 1 -- start start 2024-03-18T14:42:00.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7fec803a9640 1 -- start start 2024-03-18T14:42:00.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb979e64640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb96c00aad0 0x7fb96c04bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb979e64640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c04fa70 0x7fb96c0dc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb979e64640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96c0530a0 0x7fb96c0485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb979e64640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c0500a0 con 0x7fb96c0530a0 2024-03-18T14:42:00.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb979e64640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c04ff20 con 0x7fb96c00aad0 2024-03-18T14:42:00.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb979e64640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb96c050220 con 0x7fb96c04fa70 2024-03-18T14:42:00.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb973fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c04fa70 0x7fb96c0dc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f7f1e085640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f180d0c20 0x7f7f180cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fdac9d9b640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4004800 0x7fdab4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:00.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fdaca59c640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdab4003a00 0x7fdab4003e00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdabc003d80 tx=0x7fdab4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6115177b5b1e6661 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc037070 con 0x7fdab4003a00 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdabc02be50 con 0x7fdab4003a00 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.172+0000 7fdac959a640 1 -- 172.21.15.6:0/754053831 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc003980 con 0x7fdab4003a00 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fdacb59e640 1 -- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdab4003a00 msgr2=0x7fdab4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fdacb59e640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdab4003a00 0x7fdab4003e00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdabc003d80 tx=0x7fdab4005830 comp rx=0 tx=0).stop 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fdacb59e640 1 -- 172.21.15.6:0/754053831 shutdown_connections 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fdacb59e640 1 --2- 172.21.15.6:0/754053831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdab4003a00 0x7fdab4003e00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7fdacb59e640 1 -- 172.21.15.6:0/754053831 >> 172.21.15.6:0/754053831 conn(0x7fdac4009fb0 msgr2=0x7fdac400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fbd05805640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd0015c830 0x7fbd0015ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f180ce740 0x7f7f18158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f180ce740 0x7f7f18158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50696/0 (socket says 172.21.15.6:50696) 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f7f177fe640 1 -- 172.21.15.6:0/804325217 learned_addr learned my addr 172.21.15.6:0/804325217 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7f3bc3fff640 1 Processor -- start 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.173+0000 7f3bc3fff640 1 -- start start 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.174+0000 7f3bc3fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc050330 0x7f3bbc186850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc3fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bbc0df000 0x7f3bbc17fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc2ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc050330 0x7f3bbc186850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc2ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc050330 0x7f3bbc186850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50672/0 (socket says 172.21.15.6:50672) 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc2ffd640 1 -- 172.21.15.6:0/1875210616 learned_addr learned my addr 172.21.15.6:0/1875210616 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc3fff640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bbc0df3e0 0x7f3bbc047e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc27fc640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bbc0df000 0x7f3bbc17fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc3fff640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc007680 con 0x7f3bbc0df3e0 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc3fff640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc0073a0 con 0x7f3bbc0df000 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc3fff640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3bbc007ba0 con 0x7f3bbc050330 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f1e085640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f18162850 0x7f7f180cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb973fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c04fa70 0x7fb96c0dc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50682/0 (socket says 172.21.15.6:50682) 2024-03-18T14:42:00.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7fb973fff640 1 -- 172.21.15.6:0/3518804055 learned_addr learned my addr 172.21.15.6:0/3518804055 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.175+0000 7f3bc37fe640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bbc0df3e0 0x7f3bbc047e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f614299f640 1 Processor -- start 2024-03-18T14:42:00.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f1e085640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7f180ceb20 con 0x7f7f180d0c20 2024-03-18T14:42:00.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f1e085640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18166d50 con 0x7f7f18162850 2024-03-18T14:42:00.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f1e085640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7f180ceca0 con 0x7f7f180ce740 2024-03-18T14:42:00.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fbd05805640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd0015c830 0x7fbd0015ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56454/0 (socket says 172.21.15.6:56454) 2024-03-18T14:42:00.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fbd05805640 1 -- 172.21.15.6:0/945428660 learned_addr learned my addr 172.21.15.6:0/945428660 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec803a9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec64004800 0x7fec781fbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec803a9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec78154970 0x7fec78154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb973fff640 1 -- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb96c00aad0 msgr2=0x7fb96c04bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb973fff640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb96c00aad0 0x7fb96c04bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb973fff640 1 -- 172.21.15.6:0/3518804055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96c0530a0 msgr2=0x7fb96c0485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f17fff640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f18162850 0x7f7f180cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fa6e507e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 0x7fa6d80de640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fdacb59e640 1 -- 172.21.15.6:0/754053831 shutdown_connections 2024-03-18T14:42:00.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb979663640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96c0530a0 0x7fb96c0485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fbd05004640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd000b2870 0x7fbd0014dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fa6e507e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d80e3040 0x7fa6d803dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec803a9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec78148cf0 0x7fec78149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec803a9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fec78162660 con 0x7fec64004800 2024-03-18T14:42:00.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec803a9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fec7806ee50 con 0x7fec78148cf0 2024-03-18T14:42:00.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec803a9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fec781627e0 con 0x7fec78154970 2024-03-18T14:42:00.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb973fff640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96c0530a0 0x7fb96c0485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb973fff640 1 -- 172.21.15.6:0/3518804055 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c0dcd10 con 0x7fb96c04fa70 2024-03-18T14:42:00.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fa6e507e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6d80e3420 0x7fa6d803e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fa6e507e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d80e7a50 con 0x7fa6d80544a0 2024-03-18T14:42:00.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fa6e507e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d80e78d0 con 0x7fa6d80e3040 2024-03-18T14:42:00.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb979663640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb96c0530a0 0x7fb96c0485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:00.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f177fe640 1 -- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f18162850 msgr2=0x7f7f180cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f177fe640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f18162850 0x7f7f180cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fa6e507e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d80e7bd0 con 0x7fa6d80e3420 2024-03-18T14:42:00.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f177fe640 1 -- 172.21.15.6:0/804325217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f180d0c20 msgr2=0x7f7f180cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec7d944640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec78154970 0x7fec78154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec7d944640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec78154970 0x7fec78154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50726/0 (socket says 172.21.15.6:50726) 2024-03-18T14:42:00.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec7d944640 1 -- 172.21.15.6:0/1801494745 learned_addr learned my addr 172.21.15.6:0/1801494745 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec7e145640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec64004800 0x7fec781fbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb973fff640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c04fa70 0x7fb96c0dc7d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb968000c00 tx=0x7fb9680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fec7e946640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec78148cf0 0x7fec78149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f614299f640 1 -- start start 2024-03-18T14:42:00.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f177fe640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f180d0c20 0x7f7f180cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f177fe640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f180cd740 con 0x7f7f180ce740 2024-03-18T14:42:00.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96803cac0 con 0x7fb96c04fa70 2024-03-18T14:42:00.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fbd04803640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 0x7fbd0015e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f7f177fe640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f180ce740 0x7f7f18158830 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800d800 tx=0x7f7f0800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f3bc2ffd640 1 -- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bbc0df000 msgr2=0x7f3bbc17fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f3bc2ffd640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bbc0df000 0x7f3bbc17fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f3bc2ffd640 1 -- 172.21.15.6:0/1875210616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bbc0df3e0 msgr2=0x7f3bbc047e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f3bc2ffd640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3bbc0df3e0 0x7f3bbc047e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7f3bc2ffd640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bbc186d90 con 0x7f3bbc050330 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96c048b30 con 0x7fb96c04fa70 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96c0e1c00 con 0x7fb96c04fa70 2024-03-18T14:42:00.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f7efffff640 1 -- 172.21.15.6:0/804325217 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800b6c0 con 0x7f7f180ce740 2024-03-18T14:42:00.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb96803cc60 con 0x7fb96c04fa70 2024-03-18T14:42:00.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fa6ded9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 0x7fa6d80de640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fa6de59c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d80e3040 0x7fa6d803dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fa6ded9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 0x7fa6d80de640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56488/0 (socket says 172.21.15.6:56488) 2024-03-18T14:42:00.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fa6ded9d640 1 -- 172.21.15.6:0/1062946975 learned_addr learned my addr 172.21.15.6:0/1062946975 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fbd05004640 1 -- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 msgr2=0x7fbd0015e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fbd05004640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd000b39e0 0x7fbd0015e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f7efffff640 1 -- 172.21.15.6:0/804325217 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f08019070 con 0x7f7f180ce740 2024-03-18T14:42:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fbd05004640 1 -- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd0015c830 msgr2=0x7fbd0015ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fbd05004640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd0015c830 0x7fbd0015ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f7efffff640 1 -- 172.21.15.6:0/804325217 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800ba60 con 0x7f7f180ce740 2024-03-18T14:42:00.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fbd05004640 1 -- 172.21.15.6:0/945428660 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0015d020 con 0x7fbd000b2870 2024-03-18T14:42:00.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fbd05805640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd0015c830 0x7fbd0015ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:00.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614299f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6134038520 0x7f613417eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614299f640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6134054200 0x7f613417d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614299f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f613404d930 0x7f6134005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614299f640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6134054850 con 0x7f613404d930 2024-03-18T14:42:00.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614299f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6134052c20 con 0x7f6134038520 2024-03-18T14:42:00.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614299f640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6134053f60 con 0x7f6134054200 2024-03-18T14:42:00.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614219e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f613404d930 0x7f6134005e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614219e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f613404d930 0x7f6134005e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56494/0 (socket says 172.21.15.6:56494) 2024-03-18T14:42:00.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7f614219e640 1 -- 172.21.15.6:0/2095934981 learned_addr learned my addr 172.21.15.6:0/2095934981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fa6df59e640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6d80e3420 0x7fa6d803e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fec7e145640 1 -- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec78154970 msgr2=0x7fec78154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fec7e145640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec78154970 0x7fec78154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fec7e145640 1 -- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec78148cf0 msgr2=0x7fec78149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.177+0000 7fec7e145640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec78148cf0 0x7fec78149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fec7e145640 1 -- 172.21.15.6:0/1801494745 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec78155510 con 0x7fec64004800 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fec7e946640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec78148cf0 0x7fec78149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614199d640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6134038520 0x7f613417eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614119c640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6134054200 0x7f613417d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.176+0000 7fdacb59e640 1 -- 172.21.15.6:0/754053831 wait complete. 2024-03-18T14:42:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fdacb59e640 1 Processor -- start 2024-03-18T14:42:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fa6df59e640 1 -- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d80e3040 msgr2=0x7fa6d803dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fa6df59e640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d80e3040 0x7fa6d803dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fa6df59e640 1 -- 172.21.15.6:0/1062946975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 msgr2=0x7fa6d80de640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fa6df59e640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6d80544a0 0x7fa6d80de640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fa6df59e640 1 -- 172.21.15.6:0/1062946975 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d804ccd0 con 0x7fa6d80e3420 2024-03-18T14:42:00.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fec7e145640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec64004800 0x7fec781fbfb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec70002230 tx=0x7fec7000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f7f1e085640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f180cae00 con 0x7f7f180ce740 2024-03-18T14:42:00.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968045650 con 0x7fb96c04fa70 2024-03-18T14:42:00.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614219e640 1 -- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6134054200 msgr2=0x7f613417d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fdacb59e640 1 -- start start 2024-03-18T14:42:00.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fa6df59e640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6d80e3420 0x7fa6d803e4f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4033000 tx=0x7fa6d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f7f1e085640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f181ebe90 con 0x7f7f180ce740 2024-03-18T14:42:00.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614219e640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6134054200 0x7f613417d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614219e640 1 -- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6134038520 msgr2=0x7f613417eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fbd05004640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd000b2870 0x7fbd0014dc90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d620 tx=0x7fbcf000dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614219e640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6134038520 0x7f613417eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614219e640 1 -- 172.21.15.6:0/2095934981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613417e050 con 0x7f613404d930 2024-03-18T14:42:00.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f614199d640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6134038520 0x7f613417eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:00.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7f3bc2ffd640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc050330 0x7f3bbc186850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400c450 tx=0x7f3bb400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.178+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70013420 con 0x7fec64004800 2024-03-18T14:42:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf000b620 con 0x7fbd000b2870 2024-03-18T14:42:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb954001030 con 0x7fb96c04fa70 2024-03-18T14:42:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcf0019070 con 0x7fbd000b2870 2024-03-18T14:42:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec781557f0 con 0x7fec64004800 2024-03-18T14:42:00.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec70018070 con 0x7fec64004800 2024-03-18T14:42:00.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec781575c0 con 0x7fec64004800 2024-03-18T14:42:00.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70013800 con 0x7fec64004800 2024-03-18T14:42:00.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40052d0 con 0x7f3bbc050330 2024-03-18T14:42:00.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7f614219e640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f613404d930 0x7f6134005e20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f613c074860 tx=0x7f613c075460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb4005470 con 0x7f3bbc050330 2024-03-18T14:42:00.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4012870 con 0x7f3bbc050330 2024-03-18T14:42:00.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd00161310 con 0x7fbd000b2870 2024-03-18T14:42:00.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd001618d0 con 0x7fbd000b2870 2024-03-18T14:42:00.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf000b620 con 0x7fbd000b2870 2024-03-18T14:42:00.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c073240 con 0x7f613404d930 2024-03-18T14:42:00.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613c08e030 con 0x7f613404d930 2024-03-18T14:42:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fdacb59e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4003a00 0x7fdac417da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fdacb59e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac4034e30 0x7fdac417c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdacb59e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac4043680 0x7fdac4043b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdac9d9b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac4034e30 0x7fdac417c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdac9d9b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac4034e30 0x7fdac417c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50756/0 (socket says 172.21.15.6:50756) 2024-03-18T14:42:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdaca59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4003a00 0x7fdac417da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb968043020 con 0x7fb96c04fa70 2024-03-18T14:42:00.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4032820 con 0x7fa6d80e3420 2024-03-18T14:42:00.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d40329c0 con 0x7fa6d80e3420 2024-03-18T14:42:00.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4032df0 con 0x7fa6d80e3420 2024-03-18T14:42:00.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d804cfb0 con 0x7fa6d80e3420 2024-03-18T14:42:00.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.179+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d80deb80 con 0x7fa6d80e3420 2024-03-18T14:42:00.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdaca59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4003a00 0x7fdac417da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56498/0 (socket says 172.21.15.6:56498) 2024-03-18T14:42:00.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdac9d9b640 1 -- 172.21.15.6:0/744488828 learned_addr learned my addr 172.21.15.6:0/744488828 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdacad9d640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac4043680 0x7fdac4043b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdac4054940 con 0x7fdab4003a00 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c0736c0 con 0x7f613404d930 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7f614299f640 1 -- 172.21.15.6:0/2095934981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613417e330 con 0x7f613404d930 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7f614299f640 1 -- 172.21.15.6:0/2095934981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6134048700 con 0x7f613404d930 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdac40547c0 con 0x7fdac4043680 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdac404f550 con 0x7fdac4034e30 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fdac9d9b640 1 -- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac4043680 msgr2=0x7fdac4043b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fdac9d9b640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac4043680 0x7fdac4043b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fdac9d9b640 1 -- 172.21.15.6:0/744488828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4003a00 msgr2=0x7fdac417da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fdac9d9b640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdab4003a00 0x7fdac417da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fdac9d9b640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac40355a0 con 0x7fdac4034e30 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fb9717fa640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb95806bca0 0x7fb95806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.180+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb9680ac800 con 0x7fb96c04fa70 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fb9717fa640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7fb9580753a0 0x7fb9580777e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 --> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb96804dec0 con 0x7fb9580753a0 2024-03-18T14:42:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7f3bc3fff640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bbc0d6fb0 con 0x7f3bbc050330 2024-03-18T14:42:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7f3bc3fff640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bbc0d7570 con 0x7f3bbc050330 2024-03-18T14:42:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fdac9d9b640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac4034e30 0x7fdac417c700 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdac0004a10 tx=0x7fdac0004ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fec700139a0 con 0x7fec64004800 2024-03-18T14:42:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbce0001030 con 0x7fbd000b2870 2024-03-18T14:42:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6118001030 con 0x7f613404d930 2024-03-18T14:42:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7f7efffff640 1 -- 172.21.15.6:0/804325217 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7f0800bc00 con 0x7f7f180ce740 2024-03-18T14:42:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7f7efffff640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f0406bca0 0x7f7f0406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.181+0000 7f7efffff640 1 -- 172.21.15.6:0/804325217 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7f0808b1b0 con 0x7f7f180ce740 2024-03-18T14:42:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7f7f1e085640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7f7f18158e80 0x7f7f1806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec60001030 con 0x7fec64004800 2024-03-18T14:42:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fb978e62640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb95806bca0 0x7fb95806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fb979663640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7fb9580753a0 0x7fb9580777e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7f3bc3fff640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3bbc005710 con 0x7f3bbc050330 2024-03-18T14:42:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa6c0001030 con 0x7fa6d80e3420 2024-03-18T14:42:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7f7f17fff640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7f7f18158e80 0x7f7f1806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7f7f1e085640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f1804f5b0 con 0x7f7f18158e80 2024-03-18T14:42:00.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fb978e62640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb95806bca0 0x7fb95806e160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb974064cd0 tx=0x7fb97406a420 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb9680b3050 con 0x7fb96c04fa70 2024-03-18T14:42:00.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7f7f17fff640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7f7f18158e80 0x7f7f1806c6a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7f7f16ffd640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f0406bca0 0x7f7f0406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fb979663640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7fb9580753a0 0x7fb9580777e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fec6effd640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fec5c06bcd0 0x7fec5c06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0007bc0 con 0x7fdac4034e30 2024-03-18T14:42:00.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdac0018070 con 0x7fdac4034e30 2024-03-18T14:42:00.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.182+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac000fc00 con 0x7fdac4034e30 2024-03-18T14:42:00.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac4035880 con 0x7fdac4034e30 2024-03-18T14:42:00.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac40d7e00 con 0x7fdac4034e30 2024-03-18T14:42:00.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3bb4012a10 con 0x7f3bbc050330 2024-03-18T14:42:00.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fec7d944640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fec5c06bcd0 0x7fec5c06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fec7008b430 con 0x7fec64004800 2024-03-18T14:42:00.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7f7f16ffd640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f0406bca0 0x7f7f0406e160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f180b9a40 tx=0x7f7f10009250 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbcf000b7c0 con 0x7fbd000b2870 2024-03-18T14:42:00.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fec6effd640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7fec5c075440 0x7fec5c077880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 --> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec70020e80 con 0x7fec5c075440 2024-03-18T14:42:00.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fec70092050 con 0x7fec64004800 2024-03-18T14:42:00.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.183+0000 7f3bab7fe640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3bb006bc00 0x7f3bb006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3bb408a830 con 0x7f3bbc050330 2024-03-18T14:42:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7f3bab7fe640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7f3bb0075380 0x7f3bb00777c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3bb0077e90 con 0x7f3bb0075380 2024-03-18T14:42:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fec7d944640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fec5c06bcd0 0x7fec5c06e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fec68009860 tx=0x7fec68008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa6d403b3f0 con 0x7fa6d80e3420 2024-03-18T14:42:00.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fbcf5ffb640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbce406bdd0 0x7fbce406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f613c06f460 con 0x7f613404d930 2024-03-18T14:42:00.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3bb408f2a0 con 0x7f3bbc050330 2024-03-18T14:42:00.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fec7e946640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7fec5c075440 0x7fec5c077880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7f3bc37fe640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7f3bb0075380 0x7f3bb00777c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7f7efffff640 1 -- 172.21.15.6:0/804325217 <== osd.3 v2:172.21.15.6:6808/536857262 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f1804f5b0 con 0x7f7f18158e80 2024-03-18T14:42:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbcf008b7f0 con 0x7fbd000b2870 2024-03-18T14:42:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== osd.5 v2:172.21.15.92:6808/4136952425 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb96804dec0 con 0x7fb9580753a0 2024-03-18T14:42:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fbd04803640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbce406bdd0 0x7fbce406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f61327fc640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f612006bca0 0x7f612006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f613c105b80 con 0x7f613404d930 2024-03-18T14:42:00.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f3bc27fc640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3bb006bc00 0x7f3bb006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fec7e946640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7fec5c075440 0x7fec5c077880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.184+0000 7fa6c77fe640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6cc06bcd0 0x7fa6cc06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa6d40acb80 con 0x7fa6d80e3420 2024-03-18T14:42:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdac4035da0 con 0x7fdac4034e30 2024-03-18T14:42:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f61327fc640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6120075420 0x7f6120077860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fbcf5ffb640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7fbce4075450 0x7fbce4077890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 --> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbce4077f60 con 0x7fbce4075450 2024-03-18T14:42:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbcf008bc80 con 0x7fbd000b2870 2024-03-18T14:42:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fbd04803640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbce406bdd0 0x7fbce406e290 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbd000cc410 tx=0x7fbcf8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 --> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f613c08fe50 con 0x7f6120075420 2024-03-18T14:42:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f613c10a2a0 con 0x7f613404d930 2024-03-18T14:42:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f614119c640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6120075420 0x7f6120077860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f614199d640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f612006bca0 0x7f612006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7f3bc37fe640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7f3bb0075380 0x7f3bb00777c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7f614119c640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6120075420 0x7f6120077860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fbd05805640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7fbce4075450 0x7fbce4077890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== osd.1 v2:172.21.15.6:6809/2441154404 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec70020e80 con 0x7fec5c075440 2024-03-18T14:42:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fa6c77fe640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7fa6cc075450 0x7fa6cc077890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 --> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6d4043ec0 con 0x7fa6cc075450 2024-03-18T14:42:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa6d40b12a0 con 0x7fa6d80e3420 2024-03-18T14:42:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fa6ded9d640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6cc06bcd0 0x7fa6cc06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fa6de59c640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7fa6cc075450 0x7fa6cc077890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.185+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdac001d460 con 0x7fdac4034e30 2024-03-18T14:42:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fdabaffd640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaac06bca0 0x7fdaac06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdac008ad70 con 0x7fdac4034e30 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fdabaffd640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7fdaac075420 0x7fdaac077860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdaac077f30 con 0x7fdaac075420 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdac008d3d0 con 0x7fdac4034e30 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fdacad9d640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7fdaac075420 0x7fdaac077860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fdaca59c640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaac06bca0 0x7fdaac06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7f3bc27fc640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3bb006bc00 0x7f3bb006e0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8039000 tx=0x7f3bb800a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.186+0000 7fbd05805640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7fbce4075450 0x7fbce4077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.187+0000 7f614199d640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f612006bca0 0x7f612006e160 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6138000f60 tx=0x7f6138007e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.187+0000 7fdacad9d640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7fdaac075420 0x7fdaac077860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.187+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== osd.6 v2:172.21.15.92:6800/1041453586 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bb0077e90 con 0x7f3bb0075380 2024-03-18T14:42:00.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.187+0000 7fa6de59c640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7fa6cc075450 0x7fa6cc077890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.187+0000 7fdaca59c640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaac06bca0 0x7fdaac06e160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdabc034000 tx=0x7fdabc009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.187+0000 7fa6ded9d640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6cc06bcd0 0x7fa6cc06e190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0005880 tx=0x7fa6d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.187+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== osd.2 v2:172.21.15.6:6800/3396973445 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f613c08fe50 con 0x7f6120075420 2024-03-18T14:42:00.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.188+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== osd.0 v2:172.21.15.6:6824/3434665479 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce4077f60 con 0x7fbce4075450 2024-03-18T14:42:00.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.188+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== osd.4 v2:172.21.15.92:6816/3636494559 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaac077f30 con 0x7fdaac075420 2024-03-18T14:42:00.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.191+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== osd.7 v2:172.21.15.92:6824/1103567419 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa6d4043ec0 con 0x7fa6cc075450 2024-03-18T14:42:00.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.216+0000 7f3bc3fff640 1 -- 172.21.15.6:0/1875210616 --> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3bbc054b20 con 0x7f3bb0075380 2024-03-18T14:42:00.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.216+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 --> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb954002dd0 con 0x7fb9580753a0 2024-03-18T14:42:00.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.216+0000 7f3bab7fe640 1 -- 172.21.15.6:0/1875210616 <== osd.6 v2:172.21.15.92:6800/1041453586 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3bbc054b20 con 0x7f3bb0075380 2024-03-18T14:42:00.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.216+0000 7fb9717fa640 1 -- 172.21.15.6:0/3518804055 <== osd.5 v2:172.21.15.92:6808/4136952425 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb954002dd0 con 0x7fb9580753a0 2024-03-18T14:42:00.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7f7f1e085640 1 -- 172.21.15.6:0/804325217 --> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f180b6200 con 0x7f7f18158e80 2024-03-18T14:42:00.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7fb9580753a0 msgr2=0x7fb9580777e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7fb979e64640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7fb9580753a0 0x7fb9580777e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7f3ba97fa640 1 -- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7f3bb0075380 msgr2=0x7f3bb00777c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb95806bca0 msgr2=0x7fb95806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7fb979e64640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb95806bca0 0x7fb95806e160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb974064cd0 tx=0x7fb97406a420 comp rx=0 tx=0).stop 2024-03-18T14:42:00.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7f3ba97fa640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7f3bb0075380 0x7f3bb00777c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c04fa70 msgr2=0x7fb96c0dc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7fb979e64640 1 --2- 172.21.15.6:0/3518804055 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb96c04fa70 0x7fb96c0dc7d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb968000c00 tx=0x7fb9680046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.217+0000 7fb979663640 1 -- 172.21.15.6:0/3518804055 reap_dead start 2024-03-18T14:42:00.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f7efffff640 1 -- 172.21.15.6:0/804325217 <== osd.3 v2:172.21.15.6:6808/536857262 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7f180b6200 con 0x7f7f18158e80 2024-03-18T14:42:00.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 shutdown_connections 2024-03-18T14:42:00.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 >> 172.21.15.6:0/3518804055 conn(0x7fb96c009fb0 msgr2=0x7fb96c0daae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f3ba97fa640 1 -- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3bb006bc00 msgr2=0x7f3bb006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f3ba97fa640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3bb006bc00 0x7f3bb006e0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8039000 tx=0x7f3bb800a040 comp rx=0 tx=0).stop 2024-03-18T14:42:00.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f3ba97fa640 1 -- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc050330 msgr2=0x7f3bbc186850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f3ba97fa640 1 --2- 172.21.15.6:0/1875210616 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bbc050330 0x7f3bbc186850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400c450 tx=0x7f3bb400c910 comp rx=0 tx=0).stop 2024-03-18T14:42:00.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f3bc37fe640 1 -- 172.21.15.6:0/1875210616 reap_dead start 2024-03-18T14:42:00.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 shutdown_connections 2024-03-18T14:42:00.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7fb979e64640 1 -- 172.21.15.6:0/3518804055 wait complete. 2024-03-18T14:42:00.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f3ba97fa640 1 -- 172.21.15.6:0/1875210616 shutdown_connections 2024-03-18T14:42:00.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f3ba97fa640 1 -- 172.21.15.6:0/1875210616 >> 172.21.15.6:0/1875210616 conn(0x7f3bbc00a400 msgr2=0x7f3bbc048c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f7efdffb640 1 -- 172.21.15.6:0/804325217 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7f7f18158e80 msgr2=0x7f7f1806c6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.218+0000 7f7efdffb640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7f7f18158e80 0x7f7f1806c6a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f3ba97fa640 1 -- 172.21.15.6:0/1875210616 shutdown_connections 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f3ba97fa640 1 -- 172.21.15.6:0/1875210616 wait complete. 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f7efdffb640 1 -- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f0406bca0 msgr2=0x7f7f0406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f7efdffb640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f0406bca0 0x7f7f0406e160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f180b9a40 tx=0x7f7f10009250 comp rx=0 tx=0).stop 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f7efdffb640 1 -- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f180ce740 msgr2=0x7f7f18158830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f7efdffb640 1 --2- 172.21.15.6:0/804325217 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f180ce740 0x7f7f18158830 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800d800 tx=0x7f7f0800dcc0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f7f17fff640 1 -- 172.21.15.6:0/804325217 reap_dead start 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f7efdffb640 1 -- 172.21.15.6:0/804325217 shutdown_connections 2024-03-18T14:42:00.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.219+0000 7f7efdffb640 1 -- 172.21.15.6:0/804325217 >> 172.21.15.6:0/804325217 conn(0x7f7f18072ce0 msgr2=0x7f7f18164200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.220+0000 7f7efdffb640 1 -- 172.21.15.6:0/804325217 shutdown_connections 2024-03-18T14:42:00.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.220+0000 7f7efdffb640 1 -- 172.21.15.6:0/804325217 wait complete. 2024-03-18T14:42:00.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.220+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 --> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbce0002dd0 con 0x7fbce4075450 2024-03-18T14:42:00.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.220+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 --> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6118002dd0 con 0x7f6120075420 2024-03-18T14:42:00.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 --> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec60002dd0 con 0x7fec5c075440 2024-03-18T14:42:00.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f61327fc640 1 -- 172.21.15.6:0/2095934981 <== osd.2 v2:172.21.15.6:6800/3396973445 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6118002dd0 con 0x7f6120075420 2024-03-18T14:42:00.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7fbcf5ffb640 1 -- 172.21.15.6:0/945428660 <== osd.0 v2:172.21.15.6:6824/3434665479 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbce0002dd0 con 0x7fbce4075450 2024-03-18T14:42:00.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6120075420 msgr2=0x7f6120077860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f611ffff640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6120075420 0x7f6120077860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f612006bca0 msgr2=0x7f612006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f611ffff640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f612006bca0 0x7f612006e160 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6138000f60 tx=0x7f6138007e80 comp rx=0 tx=0).stop 2024-03-18T14:42:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f613404d930 msgr2=0x7f6134005e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f611ffff640 1 --2- 172.21.15.6:0/2095934981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f613404d930 0x7f6134005e20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f613c074860 tx=0x7f613c075460 comp rx=0 tx=0).stop 2024-03-18T14:42:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7fec6effd640 1 -- 172.21.15.6:0/1801494745 <== osd.1 v2:172.21.15.6:6809/2441154404 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fec60002dd0 con 0x7fec5c075440 2024-03-18T14:42:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.221+0000 7f614219e640 1 -- 172.21.15.6:0/2095934981 reap_dead start 2024-03-18T14:42:00.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 --> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdac40d8000 con 0x7fdaac075420 2024-03-18T14:42:00.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 shutdown_connections 2024-03-18T14:42:00.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 >> 172.21.15.6:0/2095934981 conn(0x7f6134009fb0 msgr2=0x7f61340d9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7fbce4075450 msgr2=0x7fbce4077890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fbd07268640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7fbce4075450 0x7fbce4077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fdabaffd640 1 -- 172.21.15.6:0/744488828 <== osd.4 v2:172.21.15.92:6816/3636494559 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdac40d8000 con 0x7fdaac075420 2024-03-18T14:42:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 shutdown_connections 2024-03-18T14:42:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7f611ffff640 1 -- 172.21.15.6:0/2095934981 wait complete. 2024-03-18T14:42:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 --> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa6c0002dd0 con 0x7fa6cc075450 2024-03-18T14:42:00.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbce406bdd0 msgr2=0x7fbce406e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fbd07268640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbce406bdd0 0x7fbce406e290 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbd000cc410 tx=0x7fbcf8009040 comp rx=0 tx=0).stop 2024-03-18T14:42:00.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7fec5c075440 msgr2=0x7fec5c077880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fec803a9640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7fec5c075440 0x7fec5c077880 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd000b2870 msgr2=0x7fbd0014dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fec5c06bcd0 msgr2=0x7fec5c06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fec803a9640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fec5c06bcd0 0x7fec5c06e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fec68009860 tx=0x7fec68008030 comp rx=0 tx=0).stop 2024-03-18T14:42:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7fdaac075420 msgr2=0x7fdaac077860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.222+0000 7fdacb59e640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7fdaac075420 0x7fdaac077860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec64004800 msgr2=0x7fec781fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fec803a9640 1 --2- 172.21.15.6:0/1801494745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec64004800 0x7fec781fbfb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec70002230 tx=0x7fec7000e530 comp rx=0 tx=0).stop 2024-03-18T14:42:00.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fec7e946640 1 -- 172.21.15.6:0/1801494745 reap_dead start 2024-03-18T14:42:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fbd07268640 1 --2- 172.21.15.6:0/945428660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd000b2870 0x7fbd0014dc90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d620 tx=0x7fbcf000dae0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaac06bca0 msgr2=0x7fdaac06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaac06bca0 0x7fdaac06e160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdabc034000 tx=0x7fdabc009030 comp rx=0 tx=0).stop 2024-03-18T14:42:00.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fbd05805640 1 -- 172.21.15.6:0/945428660 reap_dead start 2024-03-18T14:42:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac4034e30 msgr2=0x7fdac417c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 --2- 172.21.15.6:0/744488828 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac4034e30 0x7fdac417c700 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdac0004a10 tx=0x7fdac0004ed0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacad9d640 1 -- 172.21.15.6:0/744488828 reap_dead start 2024-03-18T14:42:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 shutdown_connections 2024-03-18T14:42:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 >> 172.21.15.6:0/945428660 conn(0x7fbd0006b1c0 msgr2=0x7fbd000af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fa6c77fe640 1 -- 172.21.15.6:0/1062946975 <== osd.7 v2:172.21.15.92:6824/1103567419 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa6c0002dd0 con 0x7fa6cc075450 2024-03-18T14:42:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 shutdown_connections 2024-03-18T14:42:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 >> 172.21.15.6:0/744488828 conn(0x7fdac4009fb0 msgr2=0x7fdac40056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 shutdown_connections 2024-03-18T14:42:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 >> 172.21.15.6:0/1801494745 conn(0x7fec780b1b90 msgr2=0x7fec780b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 shutdown_connections 2024-03-18T14:42:00.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 shutdown_connections 2024-03-18T14:42:00.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fbd07268640 1 -- 172.21.15.6:0/945428660 wait complete. 2024-03-18T14:42:00.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fdacb59e640 1 -- 172.21.15.6:0/744488828 wait complete. 2024-03-18T14:42:00.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 shutdown_connections 2024-03-18T14:42:00.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fec803a9640 1 -- 172.21.15.6:0/1801494745 wait complete. 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.223+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7fa6cc075450 msgr2=0x7fa6cc077890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7fa6cc075450 0x7fa6cc077890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6cc06bcd0 msgr2=0x7fa6cc06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa6cc06bcd0 0x7fa6cc06e190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0005880 tx=0x7fa6d0009040 comp rx=0 tx=0).stop 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6d80e3420 msgr2=0x7fa6d803e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 --2- 172.21.15.6:0/1062946975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6d80e3420 0x7fa6d803e4f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4033000 tx=0x7fa6d40046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6df59e640 1 -- 172.21.15.6:0/1062946975 reap_dead start 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 shutdown_connections 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 >> 172.21.15.6:0/1062946975 conn(0x7fa6d803aa40 msgr2=0x7fa6d800eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 shutdown_connections 2024-03-18T14:42:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.224+0000 7fa6e507e640 1 -- 172.21.15.6:0/1062946975 wait complete. 2024-03-18T14:42:00.277 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:00.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-18T14:42:00.278 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:00.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-18T14:42:00.279 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:00.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-18T14:42:00.280 INFO:teuthology.orchestra.run.smithi006.stdout:55834574860 2024-03-18T14:42:00.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-18T14:42:00.281 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:00.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-18T14:42:00.282 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:00.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-18T14:42:00.282 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:00.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-18T14:42:00.283 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:00.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-18T14:42:00.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.488+0000 7f2a6b59e640 1 Processor -- start 2024-03-18T14:42:00.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.488+0000 7f2a6b59e640 1 -- start start 2024-03-18T14:42:00.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.489+0000 7ffa09d89640 1 Processor -- start 2024-03-18T14:42:00.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.489+0000 7ffa09d89640 1 -- start start 2024-03-18T14:42:00.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6b59e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a64052c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6b59e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2a64100f10 0x7f2a64053190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6b59e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a64054240 con 0x7f2a64054520 2024-03-18T14:42:00.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6b59e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6403a3a0 con 0x7f2a64100f10 2024-03-18T14:42:00.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6b59e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6403a680 con 0x7f2a64007380 2024-03-18T14:42:00.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6ad9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2a64100f10 0x7f2a64053190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7fa37359e640 1 Processor -- start 2024-03-18T14:42:00.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7ffa09d89640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 0x7ff9fc053bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6ad9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2a64100f10 0x7f2a64053190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40090/0 (socket says 172.21.15.6:40090) 2024-03-18T14:42:00.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7ffa09d89640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9fc0e7a80 con 0x7ff9fc0e3150 2024-03-18T14:42:00.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7ffa02ffd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff9fc0e6c90 0x7ff9fc0534e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45772/0 (socket says 172.21.15.6:45772) 2024-03-18T14:42:00.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7f2a6ad9d640 1 -- 172.21.15.6:0/291854341 learned_addr learned my addr 172.21.15.6:0/291854341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7ffa02ffd640 1 -- 172.21.15.6:0/1220932229 learned_addr learned my addr 172.21.15.6:0/1220932229 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:00.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7ffa09d89640 1 -- 172.21.15.6:0/1220932229 --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9fc0071c0 con 0x7ff9fc0e6c90 2024-03-18T14:42:00.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7f2a6ad9d640 1 -- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 msgr2=0x7f2a64052c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7f2a69d9b640 1 --2- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a64052c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7fa37359e640 1 -- start start 2024-03-18T14:42:00.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7ffa09d89640 1 -- 172.21.15.6:0/1220932229 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc03aaa0 con 0x7ff9fc0e7100 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7fc9fdbff640 1 Processor -- start 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.490+0000 7fc9fdbff640 1 -- start start 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9fdbff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc9f80b39e0 0x7fc9f8163120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9fdbff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f80b2870 0x7fc9f806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9fdbff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f80b2e30 con 0x7fc9f815c830 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9fdbff640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f80af240 con 0x7fc9f80b39e0 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9fdbff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f815cc10 con 0x7fc9f80b2870 2024-03-18T14:42:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7f2a6ad9d640 1 --2- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a64052c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9f6ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc9f80b39e0 0x7fc9f8163120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9f77fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc9f815c830 0x7fc9f80b32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45786/0 (socket says 172.21.15.6:45786) 2024-03-18T14:42:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7f2a6ad9d640 1 -- 172.21.15.6:0/291854341 >> v1:172.21.15.92:6789/0 conn(0x7f2a64054520 legacy=0x7f2a64006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9f77fe640 1 -- 172.21.15.6:0/1311998633 learned_addr learned my addr 172.21.15.6:0/1311998633 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7fc9f7fff640 1 --2- 172.21.15.6:0/1311998633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f80b2870 0x7fc9f806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.491+0000 7f2a6ad9d640 1 -- 172.21.15.6:0/291854341 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6403a960 con 0x7f2a64100f10 2024-03-18T14:42:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7ffa03fff640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 0x7ff9fc053bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7f15e33f4640 1 Processor -- start 2024-03-18T14:42:00.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fc9f67fc640 1 -- 172.21.15.6:0/1311998633 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 15943528 0 0) 0x7fc9f80b2e30 con 0x7fc9f815c830 2024-03-18T14:42:00.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fc9f67fc640 1 -- 172.21.15.6:0/1311998633 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e0003520 con 0x7fc9f815c830 2024-03-18T14:42:00.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684972224 0 0) 0x7ff9fc0e7a80 con 0x7ff9fc0e3150 2024-03-18T14:42:00.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa37359e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3740cca50 0x7fa37415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa37359e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cd070 0x7fa3740cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa37359e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa374072a10 con 0x7fa37415a6c0 2024-03-18T14:42:00.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9f4003520 con 0x7ff9fc0e3150 2024-03-18T14:42:00.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa37359e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa374072cf0 con 0x7fa3740cca50 2024-03-18T14:42:00.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa37359e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa374072fd0 con 0x7fa3740cd070 2024-03-18T14:42:00.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1438996575 0 0) 0x7ff9fc0071c0 con 0x7ff9fc0e6c90 2024-03-18T14:42:00.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9fc0e7a80 con 0x7ff9fc0e6c90 2024-03-18T14:42:00.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa372d9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cd070 0x7fa3740cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa372d9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cd070 0x7fa3740cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50786/0 (socket says 172.21.15.6:50786) 2024-03-18T14:42:00.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa372d9d640 1 -- 172.21.15.6:0/3355825418 learned_addr learned my addr 172.21.15.6:0/3355825418 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa371d9b640 1 --2- 172.21.15.6:0/3355825418 >> v2:172.21.15.6:3300/0 conn(0x7fa3740cca50 0x7fa37415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fc9f7fff640 1 -- 172.21.15.6:0/1311998633 >> v2:172.21.15.92:3300/0 conn(0x7fc9f80b39e0 msgr2=0x7fc9f8163120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7f15e33f4640 1 -- start start 2024-03-18T14:42:00.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fc9f7fff640 1 --2- 172.21.15.6:0/1311998633 >> v2:172.21.15.92:3300/0 conn(0x7fc9f80b39e0 0x7fc9f8163120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fc9f7fff640 1 -- 172.21.15.6:0/1311998633 >> v1:172.21.15.92:6789/0 conn(0x7fc9f815c830 legacy=0x7fc9f80b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa372d9d640 1 -- 172.21.15.6:0/3355825418 >> v2:172.21.15.6:3300/0 conn(0x7fa3740cca50 msgr2=0x7fa37415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa372d9d640 1 --2- 172.21.15.6:0/3355825418 >> v2:172.21.15.6:3300/0 conn(0x7fa3740cca50 0x7fa37415aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa372d9d640 1 -- 172.21.15.6:0/3355825418 >> v1:172.21.15.92:6789/0 conn(0x7fa37415a6c0 legacy=0x7fa3740cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fc9f7fff640 1 -- 172.21.15.6:0/1311998633 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f8165ce0 con 0x7fc9f80b2870 2024-03-18T14:42:00.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3445774065 0 0) 0x7ff9f4003520 con 0x7ff9fc0e3150 2024-03-18T14:42:00.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 msgr2=0x7ff9fc053bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 0x7ff9fc053bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 >> v1:172.21.15.92:6789/0 conn(0x7ff9fc0e6c90 legacy=0x7ff9fc0534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f2a69d9b640 1 --2- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a64052c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:00.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc0074a0 con 0x7ff9fc0e3150 2024-03-18T14:42:00.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7ffa0404fd90 con 0x7ff9fc0e3150 2024-03-18T14:42:00.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 >> v1:172.21.15.6:6789/0 conn(0x7ff9fc0e3150 legacy=0x7ff9fc03a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 0x7ff9f4003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9f40040e0 0x7ff9f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e33f4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f15dc15a440 0x7f15dc0721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e33f4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e33f4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15dc06da60 con 0x7f15dc0b40d0 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e33f4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f15dc072b70 con 0x7f15dc15a440 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e33f4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f15dc06dd40 con 0x7f15dc072730 2024-03-18T14:42:00.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9f4004d40 0x7ff9f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc03aaa0 con 0x7ff9f4004d40 2024-03-18T14:42:00.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f2a6ad9d640 1 --2- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64100f10 0x7f2a64053190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00bb00 tx=0x7f2a5c02dfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c339d9b90c64e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f400b9f0 con 0x7ff9f40040e0 2024-03-18T14:42:00.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f400bbf0 con 0x7ff9fc0e7100 2024-03-18T14:42:00.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e1991640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f15dc0b40d0 0x7f15dc0b2870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45802/0 (socket says 172.21.15.6:45802) 2024-03-18T14:42:00.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e1991640 1 -- 172.21.15.6:0/4260849638 learned_addr learned my addr 172.21.15.6:0/4260849638 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:00.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7f15e1190640 1 --2- 172.21.15.6:0/4260849638 >> v2:172.21.15.92:3300/0 conn(0x7f15dc15a440 0x7f15dc0721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa037fe640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 0x7ff9f4003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9f40040e0 0x7ff9f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.492+0000 7fa372d9d640 1 -- 172.21.15.6:0/3355825418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3740732b0 con 0x7fa3740cd070 2024-03-18T14:42:00.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.493+0000 7fa371d9b640 1 --2- 172.21.15.6:0/3355825418 >> v2:172.21.15.6:3300/0 conn(0x7fa3740cca50 0x7fa37415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:00.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f2a6959a640 1 -- 172.21.15.6:0/291854341 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c034070 con 0x7f2a64100f10 2024-03-18T14:42:00.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f2a6959a640 1 -- 172.21.15.6:0/291854341 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c02ed80 con 0x7f2a64100f10 2024-03-18T14:42:00.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f2a6959a640 1 -- 172.21.15.6:0/291854341 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c038620 con 0x7f2a64100f10 2024-03-18T14:42:00.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15d3fff640 1 -- 172.21.15.6:0/4260849638 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149983964 0 0) 0x7f15dc06da60 con 0x7f15dc0b40d0 2024-03-18T14:42:00.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15d3fff640 1 -- 172.21.15.6:0/4260849638 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15c8005730 con 0x7f15dc0b40d0 2024-03-18T14:42:00.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15e098f640 1 --2- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7fa372d9d640 1 --2- 172.21.15.6:0/3355825418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cd070 0x7fa3740cd4c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa36800bb10 tx=0x7fa36802e030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c9abc6fd363d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7fc9f7fff640 1 --2- 172.21.15.6:0/1311998633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f80b2870 0x7fc9f806e0b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80097a0 tx=0x7fc9e802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d88564dcd92fad78 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15e1190640 1 -- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 msgr2=0x7f15dc0b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7fc9f67fc640 1 -- 172.21.15.6:0/1311998633 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8035020 con 0x7fc9f80b2870 2024-03-18T14:42:00.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15e1190640 1 --2- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc0b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15e1190640 1 -- 172.21.15.6:0/4260849638 >> v1:172.21.15.92:6789/0 conn(0x7f15dc0b40d0 legacy=0x7f15dc0b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7fc9f67fc640 1 -- 172.21.15.6:0/1311998633 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e8031810 con 0x7fc9f80b2870 2024-03-18T14:42:00.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15e1190640 1 -- 172.21.15.6:0/4260849638 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15dc06e020 con 0x7f15dc15a440 2024-03-18T14:42:00.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7f15e098f640 1 --2- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc0b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:00.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7ffa03fff640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9f4004d40 0x7ff9f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7ffa02ffd640 1 -- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 msgr2=0x7ff9f4003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9fc0e7100 0x7ff9f4003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7ffa02ffd640 1 -- 172.21.15.6:0/1220932229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9f4004d40 msgr2=0x7ff9f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9f4004d40 0x7ff9f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7ffa02ffd640 1 -- 172.21.15.6:0/1220932229 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f4003520 con 0x7ff9f40040e0 2024-03-18T14:42:00.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7fa37159a640 1 -- 172.21.15.6:0/3355825418 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36803b070 con 0x7fa3740cd070 2024-03-18T14:42:00.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fa37159a640 1 -- 172.21.15.6:0/3355825418 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa36802ede0 con 0x7fa3740cd070 2024-03-18T14:42:00.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fa37159a640 1 -- 172.21.15.6:0/3355825418 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368036660 con 0x7fa3740cd070 2024-03-18T14:42:00.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 -- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64100f10 msgr2=0x7f2a64053190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 --2- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64100f10 0x7f2a64053190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00bb00 tx=0x7f2a5c02dfd0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 -- 172.21.15.6:0/291854341 shutdown_connections 2024-03-18T14:42:00.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 --2- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64100f10 0x7f2a64053190 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 --2- 172.21.15.6:0/291854341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a64052c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 -- 172.21.15.6:0/291854341 >> 172.21.15.6:0/291854341 conn(0x7f2a64009fb0 msgr2=0x7f2a6400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fda16577640 1 Processor -- start 2024-03-18T14:42:00.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fda16577640 1 -- start start 2024-03-18T14:42:00.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fc9fdbff640 1 -- 172.21.15.6:0/1311998633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f80b2870 msgr2=0x7fc9f806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/1311998633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f80b2870 0x7fc9f806e0b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80097a0 tx=0x7fc9e802b8b0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fc9fdbff640 1 -- 172.21.15.6:0/1311998633 shutdown_connections 2024-03-18T14:42:00.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/1311998633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f80b2870 0x7fc9f806e0b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/1311998633 >> v2:172.21.15.92:3300/0 conn(0x7fc9f80b39e0 0x7fc9f8163120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fc9fdbff640 1 -- 172.21.15.6:0/1311998633 >> 172.21.15.6:0/1311998633 conn(0x7fc9f806b1c0 msgr2=0x7fc9f806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fbea0edc640 1 Processor -- start 2024-03-18T14:42:00.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7fa9c16a0640 1 Processor -- start 2024-03-18T14:42:00.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.494+0000 7fa9c16a0640 1 -- start start 2024-03-18T14:42:00.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 -- 172.21.15.6:0/291854341 shutdown_connections 2024-03-18T14:42:00.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f15e1190640 1 --2- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc15a440 0x7f15dc0721f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15d400a7c0 tx=0x7f15d402b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2642c22138a3ab15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa37359e640 1 -- 172.21.15.6:0/3355825418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cd070 msgr2=0x7fa3740cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7ffa03fff640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9f4004d40 0x7ff9f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:00.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9f40040e0 0x7ff9f4004800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8007fc0 tx=0x7ff9f4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a971c9eeb53c2570 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8007c40 con 0x7ff9f40040e0 2024-03-18T14:42:00.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa37359e640 1 --2- 172.21.15.6:0/3355825418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cd070 0x7fa3740cd4c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa36800bb10 tx=0x7fa36802e030 comp rx=0 tx=0).stop 2024-03-18T14:42:00.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f8004ac0 con 0x7ff9f40040e0 2024-03-18T14:42:00.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fc9fdbff640 1 -- 172.21.15.6:0/1311998633 shutdown_connections 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f2a6b59e640 1 -- 172.21.15.6:0/291854341 wait complete. 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fa9c16a0640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc15fe10 0x7fa9bc162840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fa9c16a0640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa9bc06c500 0x7fa9bc0cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fa9c16a0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9bc162e50 con 0x7fa9bc06c980 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9c16a0640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa9bc163130 con 0x7fa9bc06c500 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9c16a0640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa9bc163410 con 0x7fa9bc15fe10 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9ba7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa9bc06c500 0x7fa9bc0cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9ba7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa9bc06c500 0x7fa9bc0cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56518/0 (socket says 172.21.15.6:56518) 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9ba7fc640 1 -- 172.21.15.6:0/3255876411 learned_addr learned my addr 172.21.15.6:0/3255876411 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9baffd640 1 --2- 172.21.15.6:0/3255876411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc15fe10 0x7fa9bc162840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9ba7fc640 1 -- 172.21.15.6:0/3255876411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc15fe10 msgr2=0x7fa9bc162840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9ba7fc640 1 --2- 172.21.15.6:0/3255876411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc15fe10 0x7fa9bc162840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fa9ba7fc640 1 -- 172.21.15.6:0/3255876411 >> v1:172.21.15.6:6789/0 conn(0x7fa9bc06c980 legacy=0x7fa9bc0c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7ffa027fc640 1 -- 172.21.15.6:0/1220932229 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f80070d0 con 0x7ff9f40040e0 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f15d3fff640 1 -- 172.21.15.6:0/4260849638 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4034070 con 0x7f15dc15a440 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7f15d3fff640 1 -- 172.21.15.6:0/4260849638 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d4030880 con 0x7f15dc15a440 2024-03-18T14:42:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7f15e33f4640 1 -- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc15a440 msgr2=0x7f15dc0721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7f15e33f4640 1 --2- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc15a440 0x7f15dc0721f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15d400a7c0 tx=0x7f15d402b9c0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7f15e33f4640 1 -- 172.21.15.6:0/4260849638 shutdown_connections 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7f15e33f4640 1 --2- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc0b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7f15e33f4640 1 --2- 172.21.15.6:0/4260849638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc15a440 0x7f15dc0721f0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15d400a7c0 tx=0x7f15d402b9c0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7f15e33f4640 1 -- 172.21.15.6:0/4260849638 >> 172.21.15.6:0/4260849638 conn(0x7f15dc0b1570 msgr2=0x7f15dc0b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.495+0000 7fbea0edc640 1 -- start start 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fa37359e640 1 -- 172.21.15.6:0/3355825418 shutdown_connections 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fa37359e640 1 --2- 172.21.15.6:0/3355825418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cd070 0x7fa3740cd4c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fa37359e640 1 --2- 172.21.15.6:0/3355825418 >> v2:172.21.15.6:3300/0 conn(0x7fa3740cca50 0x7fa37415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fa37359e640 1 -- 172.21.15.6:0/3355825418 >> 172.21.15.6:0/3355825418 conn(0x7fa3740b23f0 msgr2=0x7fa3740b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fbea0edc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbe9c0cf080 0x7fbe9c0d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fbea0edc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c161a10 0x7fbe9c0d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fbea0edc640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9c0d20e0 con 0x7fbe9c15dfb0 2024-03-18T14:42:00.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fbea0edc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9c15e390 con 0x7fbe9c0cf080 2024-03-18T14:42:00.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fbea0edc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9c161f40 con 0x7fbe9c161a10 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fda16577640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda10159ec0 0x7fda1015e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fda16577640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda100b3400 con 0x7fda10072760 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fda16577640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1015db30 con 0x7fda100b4870 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fda16577640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fda1015e180 con 0x7fda10159ec0 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7ffa09d89640 1 -- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9f40040e0 msgr2=0x7ff9f4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7ffa09d89640 1 --2- 172.21.15.6:0/1220932229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9f40040e0 0x7ff9f4004800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8007fc0 tx=0x7ff9f4005830 comp rx=0 tx=0).stop 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7ffa03fff640 1 -- 172.21.15.6:0/1220932229 reap_dead start 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7ffa09d89640 1 -- 172.21.15.6:0/1220932229 shutdown_connections 2024-03-18T14:42:00.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7ffa09d89640 1 -- 172.21.15.6:0/1220932229 >> 172.21.15.6:0/1220932229 conn(0x7ff9fc009fa0 msgr2=0x7ff9fc00a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fbe9a59c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fbe9c15dfb0 0x7fbe9c0ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45812/0 (socket says 172.21.15.6:45812) 2024-03-18T14:42:00.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fbe9a59c640 1 -- 172.21.15.6:0/1186169892 learned_addr learned my addr 172.21.15.6:0/1186169892 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:00.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7f15e33f4640 1 -- 172.21.15.6:0/4260849638 shutdown_connections 2024-03-18T14:42:00.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7ffa09d89640 1 -- 172.21.15.6:0/1220932229 shutdown_connections 2024-03-18T14:42:00.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7f2a6b59e640 1 Processor -- start 2024-03-18T14:42:00.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fda0ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda10159ec0 0x7fda1015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fda0ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda10159ec0 0x7fda1015e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56520/0 (socket says 172.21.15.6:56520) 2024-03-18T14:42:00.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fda0ffff640 1 -- 172.21.15.6:0/962119071 learned_addr learned my addr 172.21.15.6:0/962119071 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fa9ba7fc640 1 -- 172.21.15.6:0/3255876411 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc1636f0 con 0x7fa9bc06c500 2024-03-18T14:42:00.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7f2a6b59e640 1 -- start start 2024-03-18T14:42:00.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.498+0000 7fda0ffff640 1 -- 172.21.15.6:0/962119071 >> v1:172.21.15.92:6789/0 conn(0x7fda100b4870 legacy=0x7fda10072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.498+0000 7fda0ffff640 1 -- 172.21.15.6:0/962119071 >> v1:172.21.15.6:6789/0 conn(0x7fda10072760 legacy=0x7fda1015a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:00.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.498+0000 7fda0ffff640 1 -- 172.21.15.6:0/962119071 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1015de10 con 0x7fda10159ec0 2024-03-18T14:42:00.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.498+0000 7fbe9ad9d640 1 --2- 172.21.15.6:0/1186169892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c161a10 0x7fbe9c0d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.498+0000 7fa9ba7fc640 1 --2- 172.21.15.6:0/3255876411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc0cf700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009870 tx=0x7fa9b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e97d6d70ad5c86c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7ffa09d89640 1 -- 172.21.15.6:0/1220932229 wait complete. 2024-03-18T14:42:00.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6b59e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a641a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fa9b9ffb640 1 -- 172.21.15.6:0/3255876411 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0034070 con 0x7fa9bc06c500 2024-03-18T14:42:00.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fa9b9ffb640 1 -- 172.21.15.6:0/3255876411 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b0030840 con 0x7fa9bc06c500 2024-03-18T14:42:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6b59e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64054520 0x7f2a64035ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6b59e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a64100f10 0x7f2a6404ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6b59e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2a640067b0 con 0x7f2a64100f10 2024-03-18T14:42:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6b59e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a64006630 con 0x7f2a64054520 2024-03-18T14:42:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6b59e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2a64006930 con 0x7f2a64007380 2024-03-18T14:42:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a69d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64054520 0x7f2a64035ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7ffa09d89640 1 Processor -- start 2024-03-18T14:42:00.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.496+0000 7fc9fdbff640 1 -- 172.21.15.6:0/1311998633 wait complete. 2024-03-18T14:42:00.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7fc9fdbff640 1 Processor -- start 2024-03-18T14:42:00.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.498+0000 7fc9fdbff640 1 -- start start 2024-03-18T14:42:00.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9fdbff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9f80b39e0 0x7fc9f8208ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a69d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64054520 0x7f2a64035ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40128/0 (socket says 172.21.15.6:40128) 2024-03-18T14:42:00.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a69d9b640 1 -- 172.21.15.6:0/1786045333 learned_addr learned my addr 172.21.15.6:0/1786045333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fbe9ad9d640 1 -- 172.21.15.6:0/1186169892 >> v2:172.21.15.92:3300/0 conn(0x7fbe9c0cf080 msgr2=0x7fbe9c0d25c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T14:42:00.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fbe9ad9d640 1 --2- 172.21.15.6:0/1186169892 >> v2:172.21.15.92:3300/0 conn(0x7fbe9c0cf080 0x7fbe9c0d25c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fbe9ad9d640 1 -- 172.21.15.6:0/1186169892 >> v1:172.21.15.92:6789/0 conn(0x7fbe9c15dfb0 legacy=0x7fbe9c0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fbe9ad9d640 1 -- 172.21.15.6:0/1186169892 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9c162220 con 0x7fbe9c161a10 2024-03-18T14:42:00.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.497+0000 7f15e33f4640 1 -- 172.21.15.6:0/4260849638 wait complete. 2024-03-18T14:42:00.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f15e33f4640 1 Processor -- start 2024-03-18T14:42:00.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9fdbff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f815c830 0x7fc9f815f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9fdbff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f815f600 0x7fc9f8160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fa9b9ffb640 1 -- 172.21.15.6:0/3255876411 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0030c10 con 0x7fa9bc06c500 2024-03-18T14:42:00.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9fdbff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f81662a0 con 0x7fc9f815f600 2024-03-18T14:42:00.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9fdbff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f8163660 con 0x7fc9f80b39e0 2024-03-18T14:42:00.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9fdbff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f8166580 con 0x7fc9f815c830 2024-03-18T14:42:00.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6a59c640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a641a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9f6ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f815c830 0x7fc9f815f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f2a6ad9d640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a64100f10 0x7f2a6404ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7fc9f6ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f815c830 0x7fc9f815f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50842/0 (socket says 172.21.15.6:50842) 2024-03-18T14:42:00.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fc9f6ffd640 1 -- 172.21.15.6:0/587929961 learned_addr learned my addr 172.21.15.6:0/587929961 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fc9f77fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9f80b39e0 0x7fc9f8208ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 -- 172.21.15.6:0/3255876411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 msgr2=0x7fa9bc0cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/3255876411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc0cf700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009870 tx=0x7fa9b002b980 comp rx=0 tx=0).stop 2024-03-18T14:42:00.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa37359e640 1 -- 172.21.15.6:0/3355825418 shutdown_connections 2024-03-18T14:42:00.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fc9f7fff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f815f600 0x7fc9f8160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fbe9ad9d640 1 --2- 172.21.15.6:0/1186169892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c161a10 0x7fbe9c0d2b00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009d90 tx=0x7fbe9402bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b215cf7ea74689 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 -- 172.21.15.6:0/3255876411 shutdown_connections 2024-03-18T14:42:00.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7f15e33f4640 1 -- start start 2024-03-18T14:42:00.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/3255876411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc0cf700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fc9f6ffd640 1 -- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9f80b39e0 msgr2=0x7fc9f8208ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/3255876411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc15fe10 0x7fa9bc162840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 -- 172.21.15.6:0/3255876411 >> 172.21.15.6:0/3255876411 conn(0x7fa9bc09ae00 msgr2=0x7fa9bc08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 -- 172.21.15.6:0/3255876411 shutdown_connections 2024-03-18T14:42:00.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f2a6ad9d640 1 -- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 msgr2=0x7f2a641a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f2a6ad9d640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a641a3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f2a6ad9d640 1 -- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64054520 msgr2=0x7f2a64035ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f2a6ad9d640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64054520 0x7f2a64035ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f2a6ad9d640 1 -- 172.21.15.6:0/1786045333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a64035400 con 0x7f2a64100f10 2024-03-18T14:42:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fda0ffff640 1 --2- 172.21.15.6:0/962119071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda10159ec0 0x7fda1015e610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fda08009e50 tx=0x7fda0802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb32260e0625267 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fbe9959a640 1 -- 172.21.15.6:0/1186169892 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94039070 con 0x7fbe9c161a10 2024-03-18T14:42:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fbe9959a640 1 -- 172.21.15.6:0/1186169892 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe9402ca30 con 0x7fbe9c161a10 2024-03-18T14:42:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f2a6ad9d640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a64100f10 0x7f2a6404ac80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c036000 tx=0x7f2a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7fa9c16a0640 1 -- 172.21.15.6:0/3255876411 wait complete. 2024-03-18T14:42:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.499+0000 7ffa09d89640 1 -- start start 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7ffa09d89640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9f40040e0 0x7ff9fc04a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7ffa09d89640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9fc0e4b00 0x7ff9fc0e4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa09d89640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9fc0cd5f0 0x7ff9fc0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa09d89640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc052ce0 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f15e33f4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc15d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.500+0000 7f15e33f4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc0b40d0 0x7f15dc153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e1190640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc15d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e1190640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc15d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50854/0 (socket says 172.21.15.6:50854) 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e33f4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc2023f0 0x7f15dc1e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e1190640 1 -- 172.21.15.6:0/2697976991 learned_addr learned my addr 172.21.15.6:0/2697976991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc0b4520 con 0x7f15dc2023f0 2024-03-18T14:42:00.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc0ce3d0 con 0x7f15dc0b40d0 2024-03-18T14:42:00.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f15dc0b4810 con 0x7f15dc072730 2024-03-18T14:42:00.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fda0effd640 1 -- 172.21.15.6:0/962119071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0802c9f0 con 0x7fda10159ec0 2024-03-18T14:42:00.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fda0effd640 1 -- 172.21.15.6:0/962119071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0802cb90 con 0x7fda10159ec0 2024-03-18T14:42:00.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fda0effd640 1 -- 172.21.15.6:0/962119071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080343f0 con 0x7fda10159ec0 2024-03-18T14:42:00.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e098f640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc0b40d0 0x7f15dc153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fa37359e640 1 -- 172.21.15.6:0/3355825418 wait complete. 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa09d89640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc053050 con 0x7ff9fc0cd5f0 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa09d89640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc006ee0 con 0x7ff9f40040e0 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fc9f6ffd640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9f80b39e0 0x7fc9f8208ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fc9f6ffd640 1 -- 172.21.15.6:0/587929961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f815f600 msgr2=0x7fc9f8160ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fc9f6ffd640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f815f600 0x7fc9f8160ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fc9f6ffd640 1 -- 172.21.15.6:0/587929961 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f82094a0 con 0x7fc9f815c830 2024-03-18T14:42:00.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fc9f7fff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f815f600 0x7fc9f8160ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:00.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fc9f77fe640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9f80b39e0 0x7fc9f8208ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:00.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fbe9959a640 1 -- 172.21.15.6:0/1186169892 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9402ce00 con 0x7fbe9c161a10 2024-03-18T14:42:00.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa03fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9fc0cd5f0 0x7ff9fc0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa03fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9fc0cd5f0 0x7ff9fc0cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40158/0 (socket says 172.21.15.6:40158) 2024-03-18T14:42:00.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa03fff640 1 -- 172.21.15.6:0/1387870341 learned_addr learned my addr 172.21.15.6:0/1387870341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa037fe640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9f40040e0 0x7ff9fc04a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9fc0e4b00 0x7ff9fc0e4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f2a5affd640 1 -- 172.21.15.6:0/1786045333 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c0375c0 con 0x7f2a64100f10 2024-03-18T14:42:00.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fbea0edc640 1 -- 172.21.15.6:0/1186169892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c161a10 msgr2=0x7fbe9c0d2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f2a6b59e640 1 -- 172.21.15.6:0/1786045333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a640363e0 con 0x7f2a64100f10 2024-03-18T14:42:00.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f2a6b59e640 1 -- 172.21.15.6:0/1786045333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a640356a0 con 0x7f2a64100f10 2024-03-18T14:42:00.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f2a5affd640 1 -- 172.21.15.6:0/1786045333 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c037780 con 0x7f2a64100f10 2024-03-18T14:42:00.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fc9f6ffd640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f815c830 0x7fc9f815f0c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00e9e0 tx=0x7fc9ec00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7ffa02ffd640 1 -- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9f40040e0 msgr2=0x7ff9fc04a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9f40040e0 0x7ff9fc04a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7ffa02ffd640 1 -- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9fc0cd5f0 msgr2=0x7ff9fc0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9fc0cd5f0 0x7ff9fc0cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7ffa02ffd640 1 -- 172.21.15.6:0/1387870341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc0e5830 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fc9dffff640 1 -- 172.21.15.6:0/587929961 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec017070 con 0x7fc9f815c830 2024-03-18T14:42:00.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fda16577640 1 -- 172.21.15.6:0/962119071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda10159ec0 msgr2=0x7fda1015e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fda16577640 1 --2- 172.21.15.6:0/962119071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda10159ec0 0x7fda1015e610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fda08009e50 tx=0x7fda0802be50 comp rx=0 tx=0).stop 2024-03-18T14:42:00.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fda16577640 1 -- 172.21.15.6:0/962119071 shutdown_connections 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fda16577640 1 --2- 172.21.15.6:0/962119071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda10159ec0 0x7fda1015e610 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fda16577640 1 -- 172.21.15.6:0/962119071 >> 172.21.15.6:0/962119071 conn(0x7fda100b1b90 msgr2=0x7fda100b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fda16577640 1 -- 172.21.15.6:0/962119071 shutdown_connections 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fda16577640 1 -- 172.21.15.6:0/962119071 wait complete. 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f815fe30 con 0x7fc9f815c830 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f81603f0 con 0x7fc9f815c830 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fc9dffff640 1 -- 172.21.15.6:0/587929961 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9ec00c9d0 con 0x7fc9f815c830 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7fc9dffff640 1 -- 172.21.15.6:0/587929961 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec00cdc0 con 0x7fc9f815c830 2024-03-18T14:42:00.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7f2a5affd640 1 -- 172.21.15.6:0/1786045333 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c037b70 con 0x7f2a64100f10 2024-03-18T14:42:00.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7fa37359e640 1 Processor -- start 2024-03-18T14:42:00.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7fa37359e640 1 -- start start 2024-03-18T14:42:00.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7ffa02ffd640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9fc0e4b00 0x7ff9fc0e4f80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8001eb0 tx=0x7ff9f80018f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7f2a6b59e640 1 -- 172.21.15.6:0/1786045333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a50005290 con 0x7f2a64100f10 2024-03-18T14:42:00.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7ff9effff640 1 -- 172.21.15.6:0/1387870341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8007c40 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7ff9effff640 1 -- 172.21.15.6:0/1387870341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f8042020 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7ff9effff640 1 -- 172.21.15.6:0/1387870341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8015560 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1186169892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c161a10 0x7fbe9c0d2b00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009d90 tx=0x7fbe9402bd00 comp rx=0 tx=0).stop 2024-03-18T14:42:00.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fda16577640 1 Processor -- start 2024-03-18T14:42:00.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e098f640 1 -- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 msgr2=0x7f15dc15d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e098f640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc15d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fbea0edc640 1 -- 172.21.15.6:0/1186169892 shutdown_connections 2024-03-18T14:42:00.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1186169892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c161a10 0x7fbe9c0d2b00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1186169892 >> v2:172.21.15.92:3300/0 conn(0x7fbe9c0cf080 0x7fbe9c0d25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fbea0edc640 1 -- 172.21.15.6:0/1186169892 >> 172.21.15.6:0/1186169892 conn(0x7fbe9c06c850 msgr2=0x7fbe9c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e098f640 1 -- 172.21.15.6:0/2697976991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc2023f0 msgr2=0x7f15dc1e2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e098f640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc2023f0 0x7f15dc1e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.501+0000 7f15e098f640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15dc1573d0 con 0x7f15dc0b40d0 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7f15e098f640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc0b40d0 0x7f15dc153460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15d800cec0 tx=0x7f15d800a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7f15d1ffb640 1 -- 172.21.15.6:0/2697976991 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d801a070 con 0x7f15dc0b40d0 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7f15d1ffb640 1 -- 172.21.15.6:0/2697976991 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d800d840 con 0x7f15dc0b40d0 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.502+0000 7f15d1ffb640 1 -- 172.21.15.6:0/2697976991 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d800dc10 con 0x7f15dc0b40d0 2024-03-18T14:42:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fda16577640 1 -- start start 2024-03-18T14:42:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc0e5b10 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc0e60d0 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa37359e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cca50 0x7fa3740c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa37359e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740cd070 0x7fa37414ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa37359e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa37415a6c0 0x7fa3741efa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15dc1576b0 con 0x7f15dc0b40d0 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15dc157c70 con 0x7f15dc0b40d0 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fbea0edc640 1 -- 172.21.15.6:0/1186169892 shutdown_connections 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7fa9c16a0640 1 Processor -- start 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.503+0000 7fa9c16a0640 1 -- start start 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa9c16a0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc2005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa9baffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc2005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa9baffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc2005c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56556/0 (socket says 172.21.15.6:56556) 2024-03-18T14:42:00.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa9baffd640 1 -- 172.21.15.6:0/2778945767 learned_addr learned my addr 172.21.15.6:0/2778945767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9bc15fe10 0x7fa9bc0b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d8005290 con 0x7fc9f815c830 2024-03-18T14:42:00.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7f2a5affd640 1 -- 172.21.15.6:0/1786045333 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2a5c02ec40 con 0x7f2a64100f10 2024-03-18T14:42:00.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa371d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740cd070 0x7fa37414ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa371d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740cd070 0x7fa37414ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40180/0 (socket says 172.21.15.6:40180) 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.504+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc2040a0 0x7fa9bc1591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9ba7fc640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9bc15fe10 0x7fa9bc0b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc0cfc40 con 0x7fa9bc06c500 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1601f0 con 0x7fa9bc15fe10 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa9bc0cfdc0 con 0x7fa9bc2040a0 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9bb7fe640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc2040a0 0x7fa9bc1591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa371d9b640 1 -- 172.21.15.6:0/1855623930 learned_addr learned my addr 172.21.15.6:0/1855623930 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa37415e6c0 con 0x7fa37415a6c0 2024-03-18T14:42:00.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa37415e540 con 0x7fa3740cd070 2024-03-18T14:42:00.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa37415e840 con 0x7fa3740cca50 2024-03-18T14:42:00.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7f2a5affd640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2a4c06bca0 0x7f2a4c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fda16577640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda10072760 0x7fda1014a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa372d9d640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa37415a6c0 0x7fa3741efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa372d9d640 1 -- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cca50 msgr2=0x7fa3740c7910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:42:00.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa372d9d640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cca50 0x7fa3740c7910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa372d9d640 1 -- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740cd070 msgr2=0x7fa37414ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa372d9d640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740cd070 0x7fa37414ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa372d9d640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3740c7e50 con 0x7fa37415a6c0 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa372d9d640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa37415a6c0 0x7fa3741efa30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa368035000 tx=0x7fa368004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7f2a5affd640 1 -- 172.21.15.6:0/1786045333 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2a5c0ad2b0 con 0x7f2a64100f10 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9ba7fc640 1 -- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc2040a0 msgr2=0x7fa9bc1591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9ba7fc640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc2040a0 0x7fa9bc1591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9ba7fc640 1 -- 172.21.15.6:0/2778945767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 msgr2=0x7fa9bc2005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9ba7fc640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc2005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fa9ba7fc640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc204580 con 0x7fa9bc15fe10 2024-03-18T14:42:00.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa9ba7fc640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9bc15fe10 0x7fa9bc0b67d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0033000 tx=0x7fa9b0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa9a37fe640 1 -- 172.21.15.6:0/2778945767 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0048070 con 0x7fa9bc15fe10 2024-03-18T14:42:00.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa9a37fe640 1 -- 172.21.15.6:0/2778945767 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b0030640 con 0x7fa9bc15fe10 2024-03-18T14:42:00.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc204860 con 0x7fa9bc15fe10 2024-03-18T14:42:00.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa9a37fe640 1 -- 172.21.15.6:0/2778945767 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0034040 con 0x7fa9bc15fe10 2024-03-18T14:42:00.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc204e20 con 0x7fa9bc15fe10 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fda16577640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda100b4870 0x7fda1014c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fda0ffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda10072760 0x7fda1014a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fda0ffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda10072760 0x7fda1014a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40186/0 (socket says 172.21.15.6:40186) 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.505+0000 7fda0ffff640 1 -- 172.21.15.6:0/972162030 learned_addr learned my addr 172.21.15.6:0/972162030 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fda16577640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda10159ec0 0x7fda1014c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fda0f7fe640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda100b4870 0x7fda1014c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda100b2e90 con 0x7fda100b4870 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda10068430 con 0x7fda10072760 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fda14b14640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda10159ec0 0x7fda1014c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fda100b3010 con 0x7fda10159ec0 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fda0f7fe640 1 -- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda10159ec0 msgr2=0x7fda1014c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7fda0f7fe640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda10159ec0 0x7fda1014c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7fda0f7fe640 1 -- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda10072760 msgr2=0x7fda1014a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7fda0f7fe640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda10072760 0x7fda1014a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7fda0f7fe640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1014d360 con 0x7fda100b4870 2024-03-18T14:42:00.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc0052b0 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7fda0f7fe640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda100b4870 0x7fda1014c490 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fda0400b3e0 tx=0x7fda0400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7f15d1ffb640 1 -- 172.21.15.6:0/2697976991 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f15d801d070 con 0x7f15dc0b40d0 2024-03-18T14:42:00.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9bc04f5b0 con 0x7fa9bc15fe10 2024-03-18T14:42:00.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7f15d1ffb640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f15c006bca0 0x7f15c006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7f2a6a59c640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2a4c06bca0 0x7f2a4c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7f15d1ffb640 1 -- 172.21.15.6:0/2697976991 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f15d808b4c0 con 0x7f15dc0b40d0 2024-03-18T14:42:00.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15dc04f590 con 0x7f15dc0b40d0 2024-03-18T14:42:00.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7f15e1190640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f15c006bca0 0x7f15c006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.507+0000 7fc9dffff640 1 -- 172.21.15.6:0/587929961 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc9ec01a070 con 0x7fc9f815c830 2024-03-18T14:42:00.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fa371d9b640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740cd070 0x7fa37414ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:00.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.506+0000 7fbea0edc640 1 -- 172.21.15.6:0/1186169892 wait complete. 2024-03-18T14:42:00.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fa9a37fe640 1 -- 172.21.15.6:0/2778945767 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa9b004b070 con 0x7fa9bc15fe10 2024-03-18T14:42:00.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7f2a6a59c640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2a4c06bca0 0x7f2a4c06e160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c0507b0 tx=0x7f2a6c068f20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fc9dffff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc9e406bcd0 0x7fc9e406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fc9dffff640 1 -- 172.21.15.6:0/587929961 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc9ec08acf0 con 0x7fc9f815c830 2024-03-18T14:42:00.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fc9f77fe640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc9e406bcd0 0x7fc9e406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7f15e1190640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f15c006bca0 0x7f15c006e160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15d4030fd0 tx=0x7f15d403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fbea0edc640 1 Processor -- start 2024-03-18T14:42:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fa9a37fe640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa9a806bca0 0x7fa9a806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fbea0edc640 1 -- start start 2024-03-18T14:42:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fa9a37fe640 1 -- 172.21.15.6:0/2778945767 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa9b00acd40 con 0x7fa9bc15fe10 2024-03-18T14:42:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fa9baffd640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa9a806bca0 0x7fa9a806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.508+0000 7fc9f77fe640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc9e406bcd0 0x7fc9e406e190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0001560 tx=0x7fc9f00019f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fa9baffd640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa9a806bca0 0x7fa9a806e160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40094c0 tx=0x7fa9b400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbea0edc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c0cf080 0x7fbe9c0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbea0edc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c15dfb0 0x7fbe9c0cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbea0edc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe9c161a10 0x7fbe9c0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbea0edc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c0d1010 con 0x7fbe9c0cf080 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbea0edc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c1627e0 con 0x7fbe9c161a10 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbea0edc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9c0d1190 con 0x7fbe9c15dfb0 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbe9a59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c0cf080 0x7fbe9c0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbe9a59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c0cf080 0x7fbe9c0b5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56598/0 (socket says 172.21.15.6:56598) 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbe9a59c640 1 -- 172.21.15.6:0/1909275822 learned_addr learned my addr 172.21.15.6:0/1909275822 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbe99d9b640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c15dfb0 0x7fbe9c0cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.509+0000 7fbe9ad9d640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe9c161a10 0x7fbe9c0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.510+0000 7fbe9a59c640 1 -- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c15dfb0 msgr2=0x7fbe9c0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.510+0000 7fbe9a59c640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c15dfb0 0x7fbe9c0cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.510+0000 7fbe9a59c640 1 -- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe9c161a10 msgr2=0x7fbe9c0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.510+0000 7fbe9a59c640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe9c161a10 0x7fbe9c0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.510+0000 7fbe9a59c640 1 -- 172.21.15.6:0/1909275822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9c0b5970 con 0x7fbe9c0cf080 2024-03-18T14:42:00.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.510+0000 7fbe9ad9d640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe9c161a10 0x7fbe9c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:00.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.510+0000 7fbe9a59c640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c0cf080 0x7fbe9c0b5010 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c780 tx=0x7fbe8c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.511+0000 7f2a5affd640 1 -- 172.21.15.6:0/1786045333 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2a5c07bad0 con 0x7f2a64100f10 2024-03-18T14:42:00.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.511+0000 7fbe8affd640 1 -- 172.21.15.6:0/1909275822 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c017070 con 0x7fbe9c0cf080 2024-03-18T14:42:00.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.511+0000 7fda0cff9640 1 -- 172.21.15.6:0/972162030 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04015030 con 0x7fda100b4870 2024-03-18T14:42:00.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.511+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9c0b5c50 con 0x7fbe9c0cf080 2024-03-18T14:42:00.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.511+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9c0b6210 con 0x7fbe9c0cf080 2024-03-18T14:42:00.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.511+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1014d640 con 0x7fda100b4870 2024-03-18T14:42:00.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1014dc00 con 0x7fda100b4870 2024-03-18T14:42:00.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fda0cff9640 1 -- 172.21.15.6:0/972162030 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda04004020 con 0x7fda100b4870 2024-03-18T14:42:00.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fda0cff9640 1 -- 172.21.15.6:0/972162030 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04004450 con 0x7fda100b4870 2024-03-18T14:42:00.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fa362ffd640 1 -- 172.21.15.6:0/1855623930 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36803b050 con 0x7fa37415a6c0 2024-03-18T14:42:00.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fa362ffd640 1 -- 172.21.15.6:0/1855623930 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa368047070 con 0x7fa37415a6c0 2024-03-18T14:42:00.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fa362ffd640 1 -- 172.21.15.6:0/1855623930 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368040810 con 0x7fa37415a6c0 2024-03-18T14:42:00.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7ff9effff640 1 -- 172.21.15.6:0/1387870341 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff9f8007030 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3740c80d0 con 0x7fa37415a6c0 2024-03-18T14:42:00.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3740c8690 con 0x7fa37415a6c0 2024-03-18T14:42:00.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fbe8affd640 1 -- 172.21.15.6:0/1909275822 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8c00ce30 con 0x7fbe9c0cf080 2024-03-18T14:42:00.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.512+0000 7fbe8affd640 1 -- 172.21.15.6:0/1909275822 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c021060 con 0x7fbe9c0cf080 2024-03-18T14:42:00.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.513+0000 7ff9effff640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9e806bca0 0x7ff9e806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.513+0000 7ff9effff640 1 -- 172.21.15.6:0/1387870341 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff9f80ad2e0 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.513+0000 7f15d1ffb640 1 -- 172.21.15.6:0/2697976991 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f15d8059600 con 0x7f15dc0b40d0 2024-03-18T14:42:00.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.513+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f0005290 con 0x7fda100b4870 2024-03-18T14:42:00.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.513+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe80005290 con 0x7fbe9c0cf080 2024-03-18T14:42:00.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.513+0000 7ffa037fe640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9e806bca0 0x7ff9e806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.514+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3740c88d0 con 0x7fa37415a6c0 2024-03-18T14:42:00.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.514+0000 7ffa037fe640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9e806bca0 0x7ff9e806e160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffa04069700 tx=0x7ffa04074a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.515+0000 7fc9dffff640 1 -- 172.21.15.6:0/587929961 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc9ec058e30 con 0x7fc9f815c830 2024-03-18T14:42:00.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.517+0000 7ff9effff640 1 -- 172.21.15.6:0/1387870341 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff9f804c030 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.518+0000 7fa9a37fe640 1 -- 172.21.15.6:0/2778945767 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa9b007af10 con 0x7fa9bc15fe10 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.521+0000 7fbe8affd640 1 -- 172.21.15.6:0/1909275822 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbe8c01a070 con 0x7fbe9c0cf080 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.521+0000 7fbe8affd640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe7c06bbd0 0x7fbe7c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.521+0000 7fbe8affd640 1 -- 172.21.15.6:0/1909275822 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbe8c08ad70 con 0x7fbe9c0cf080 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.521+0000 7fa362ffd640 1 -- 172.21.15.6:0/1855623930 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3680409b0 con 0x7fa37415a6c0 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.520+0000 7fda0cff9640 1 -- 172.21.15.6:0/972162030 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fda04018070 con 0x7fda100b4870 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.520+0000 7fda0cff9640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9fc06bbd0 0x7fd9fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.520+0000 7fda0cff9640 1 -- 172.21.15.6:0/972162030 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fda0408afc0 con 0x7fda100b4870 2024-03-18T14:42:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.521+0000 7fda0ffff640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9fc06bbd0 0x7fd9fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.522+0000 7fa362ffd640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa35406bca0 0x7fa35406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:00.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.522+0000 7fa362ffd640 1 -- 172.21.15.6:0/1855623930 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa3680b0370 con 0x7fa37415a6c0 2024-03-18T14:42:00.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.522+0000 7fa37259c640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa35406bca0 0x7fa35406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.522+0000 7fbe99d9b640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe7c06bbd0 0x7fbe7c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:00.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.523+0000 7fda0ffff640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9fc06bbd0 0x7fd9fc06e090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fda08000f80 tx=0x7fda08030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.523+0000 7fa37259c640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa35406bca0 0x7fa35406e160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa364003c50 tx=0x7fa3640079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.523+0000 7fbe99d9b640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe7c06bbd0 0x7fbe7c06e090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c150250 tx=0x7fbe900090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:00.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.525+0000 7fda0cff9640 1 -- 172.21.15.6:0/972162030 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fda04059750 con 0x7fda100b4870 2024-03-18T14:42:00.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.531+0000 7fbe8affd640 1 -- 172.21.15.6:0/1909275822 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbe8c058f40 con 0x7fbe9c0cf080 2024-03-18T14:42:00.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.537+0000 7fa362ffd640 1 -- 172.21.15.6:0/1855623930 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3740c7e50 con 0x7fa37415a6c0 2024-03-18T14:42:00.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.937+0000 7f2a6b59e640 1 -- 172.21.15.6:0/1786045333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f2a50005580 con 0x7f2a64100f10 2024-03-18T14:42:00.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.937+0000 7f2a5affd640 1 -- 172.21.15.6:0/1786045333 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2a5c07b3d0 con 0x7f2a64100f10 2024-03-18T14:42:00.939 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:00.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.940+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f15dc1e9df0 con 0x7f15dc0b40d0 2024-03-18T14:42:00.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.940+0000 7f15d1ffb640 1 -- 172.21.15.6:0/2697976991 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f15d805d6a0 con 0x7f15dc0b40d0 2024-03-18T14:42:00.943 INFO:teuthology.orchestra.run.smithi006.stdout:55834574857 2024-03-18T14:42:00.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 -- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2a4c06bca0 msgr2=0x7f2a4c06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2a4c06bca0 0x7f2a4c06e160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c0507b0 tx=0x7f2a6c068f20 comp rx=0 tx=0).stop 2024-03-18T14:42:00.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 -- 172.21.15.6:0/1786045333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a64100f10 msgr2=0x7f2a6404ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a64100f10 0x7f2a6404ac80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c036000 tx=0x7f2a5c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 -- 172.21.15.6:0/1786045333 shutdown_connections 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2a4c06bca0 0x7f2a4c06e160 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a64100f10 0x7f2a6404ac80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a64054520 0x7f2a64035ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 --2- 172.21.15.6:0/1786045333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a64007380 0x7f2a641a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.941+0000 7f2a58ff9640 1 -- 172.21.15.6:0/1786045333 >> 172.21.15.6:0/1786045333 conn(0x7f2a64009fb0 msgr2=0x7f2a64101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.942+0000 7f2a58ff9640 1 -- 172.21.15.6:0/1786045333 shutdown_connections 2024-03-18T14:42:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.942+0000 7f2a58ff9640 1 -- 172.21.15.6:0/1786045333 wait complete. 2024-03-18T14:42:00.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.943+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f15c006bca0 msgr2=0x7f15c006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.943+0000 7f15e33f4640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f15c006bca0 0x7f15c006e160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15d4030fd0 tx=0x7f15d403b040 comp rx=0 tx=0).stop 2024-03-18T14:42:00.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.943+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc0b40d0 msgr2=0x7f15dc153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.943+0000 7f15e33f4640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc0b40d0 0x7f15dc153460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15d800cec0 tx=0x7f15d800a710 comp rx=0 tx=0).stop 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 shutdown_connections 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f15c006bca0 0x7f15c006e160 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc2023f0 0x7f15dc1e2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15dc0b40d0 0x7f15dc153460 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 --2- 172.21.15.6:0/2697976991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15dc072730 0x7f15dc15d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 >> 172.21.15.6:0/2697976991 conn(0x7f15dc0b1570 msgr2=0x7f15dc06edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 shutdown_connections 2024-03-18T14:42:00.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.945+0000 7f15e33f4640 1 -- 172.21.15.6:0/2697976991 wait complete. 2024-03-18T14:42:00.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.947+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc9d8005580 con 0x7fc9f815c830 2024-03-18T14:42:00.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.948+0000 7fc9dffff640 1 -- 172.21.15.6:0/587929961 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc9ec05ced0 con 0x7fc9f815c830 2024-03-18T14:42:00.949 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:00.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.949+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ff9fc052760 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.950+0000 7ff9effff640 1 -- 172.21.15.6:0/1387870341 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff9f807bb00 con 0x7ff9fc0e4b00 2024-03-18T14:42:00.951 INFO:teuthology.orchestra.run.smithi006.stdout:55834574857 2024-03-18T14:42:00.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa9bc165780 con 0x7fa9bc15fe10 2024-03-18T14:42:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc9e406bcd0 msgr2=0x7fc9e406e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc9e406bcd0 0x7fc9e406e190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0001560 tx=0x7fc9f00019f0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f815c830 msgr2=0x7fc9f815f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f815c830 0x7fc9f815f0c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00e9e0 tx=0x7fc9ec00eea0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 shutdown_connections 2024-03-18T14:42:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fa9a37fe640 1 -- 172.21.15.6:0/2778945767 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa9b007efb0 con 0x7fa9bc15fe10 2024-03-18T14:42:00.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc9e406bcd0 0x7fc9e406e190 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0001560 tx=0x7fc9f00019f0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9f815f600 0x7fc9f8160ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9f815c830 0x7fc9f815f0c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 --2- 172.21.15.6:0/587929961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9f80b39e0 0x7fc9f8208ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.951+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 >> 172.21.15.6:0/587929961 conn(0x7fc9f806b1c0 msgr2=0x7fc9f80b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.954 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:00.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.953+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 shutdown_connections 2024-03-18T14:42:00.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.953+0000 7fc9fdbff640 1 -- 172.21.15.6:0/587929961 wait complete. 2024-03-18T14:42:00.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.953+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd9f0005850 con 0x7fda100b4870 2024-03-18T14:42:00.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.953+0000 7fda0cff9640 1 -- 172.21.15.6:0/972162030 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fda04059050 con 0x7fda100b4870 2024-03-18T14:42:00.956 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:00.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.954+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9e806bca0 msgr2=0x7ff9e806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.954+0000 7ffa09d89640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9e806bca0 0x7ff9e806e160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffa04069700 tx=0x7ffa04074a80 comp rx=0 tx=0).stop 2024-03-18T14:42:00.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.954+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9fc0e4b00 msgr2=0x7ff9fc0e4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7ffa09d89640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9fc0e4b00 0x7ff9fc0e4f80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8001eb0 tx=0x7ff9f80018f0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa9a806bca0 msgr2=0x7fa9a806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa9a806bca0 0x7fa9a806e160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40094c0 tx=0x7fa9b400b040 comp rx=0 tx=0).stop 2024-03-18T14:42:00.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9bc15fe10 msgr2=0x7fa9bc0b67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9bc15fe10 0x7fa9bc0b67d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0033000 tx=0x7fa9b0004730 comp rx=0 tx=0).stop 2024-03-18T14:42:00.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 shutdown_connections 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 shutdown_connections 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7ffa09d89640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9e806bca0 0x7ff9e806e160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7ffa09d89640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9fc0cd5f0 0x7ff9fc0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7ffa09d89640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9fc0e4b00 0x7ff9fc0e4f80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7ffa09d89640 1 --2- 172.21.15.6:0/1387870341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9f40040e0 0x7ff9fc04a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 >> 172.21.15.6:0/1387870341 conn(0x7ff9fc009fa0 msgr2=0x7ff9fc03a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa9a806bca0 0x7fa9a806e160 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9bc2040a0 0x7fa9bc1591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.956+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 shutdown_connections 2024-03-18T14:42:00.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.956+0000 7ffa09d89640 1 -- 172.21.15.6:0/1387870341 wait complete. 2024-03-18T14:42:00.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9bc15fe10 0x7fa9bc0b67d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 --2- 172.21.15.6:0/2778945767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9bc06c500 0x7fa9bc2005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.955+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 >> 172.21.15.6:0/2778945767 conn(0x7fa9bc09ae00 msgr2=0x7fa9bc0d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.956+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 shutdown_connections 2024-03-18T14:42:00.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.956+0000 7fa9c16a0640 1 -- 172.21.15.6:0/2778945767 wait complete. 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.957+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9fc06bbd0 msgr2=0x7fd9fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.957+0000 7fda16577640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9fc06bbd0 0x7fd9fc06e090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fda08000f80 tx=0x7fda08030000 comp rx=0 tx=0).stop 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.957+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda100b4870 msgr2=0x7fda1014c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.957+0000 7fda16577640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda100b4870 0x7fda1014c490 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fda0400b3e0 tx=0x7fda0400b8a0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 shutdown_connections 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9fc06bbd0 0x7fd9fc06e090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda10159ec0 0x7fda1014c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda100b4870 0x7fda1014c490 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 --2- 172.21.15.6:0/972162030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda10072760 0x7fda1014a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 >> 172.21.15.6:0/972162030 conn(0x7fda100b1b90 msgr2=0x7fda100b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 shutdown_connections 2024-03-18T14:42:00.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.959+0000 7fda16577640 1 -- 172.21.15.6:0/972162030 wait complete. 2024-03-18T14:42:00.962 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.2 2024-03-18T14:42:00.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.961+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbe80005850 con 0x7fbe9c0cf080 2024-03-18T14:42:00.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.962+0000 7fbe8affd640 1 -- 172.21.15.6:0/1909275822 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbe8c05cfe0 con 0x7fbe9c0cf080 2024-03-18T14:42:00.963 INFO:teuthology.orchestra.run.smithi006.stdout:55834574857 2024-03-18T14:42:00.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.965+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe7c06bbd0 msgr2=0x7fbe7c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.965+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe7c06bbd0 0x7fbe7c06e090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c150250 tx=0x7fbe900090e0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.965+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c0cf080 msgr2=0x7fbe9c0b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.965+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c0cf080 0x7fbe9c0b5010 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c780 tx=0x7fbe8c00cc40 comp rx=0 tx=0).stop 2024-03-18T14:42:00.968 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.3 2024-03-18T14:42:00.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 shutdown_connections 2024-03-18T14:42:00.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe7c06bbd0 0x7fbe7c06e090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe9c161a10 0x7fbe9c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe9c15dfb0 0x7fbe9c0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 --2- 172.21.15.6:0/1909275822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c0cf080 0x7fbe9c0b5010 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 >> 172.21.15.6:0/1909275822 conn(0x7fbe9c06c850 msgr2=0x7fbe9c06cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa3740c5dc0 con 0x7fa37415a6c0 2024-03-18T14:42:00.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 shutdown_connections 2024-03-18T14:42:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fbea0edc640 1 -- 172.21.15.6:0/1909275822 wait complete. 2024-03-18T14:42:00.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.966+0000 7fa362ffd640 1 -- 172.21.15.6:0/1855623930 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa36807eb00 con 0x7fa37415a6c0 2024-03-18T14:42:00.970 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa35406bca0 msgr2=0x7fa35406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa35406bca0 0x7fa35406e160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa364003c50 tx=0x7fa3640079b0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa37415a6c0 msgr2=0x7fa3741efa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa37415a6c0 0x7fa3741efa30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa368035000 tx=0x7fa368004730 comp rx=0 tx=0).stop 2024-03-18T14:42:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 shutdown_connections 2024-03-18T14:42:00.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa35406bca0 0x7fa35406e160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa37415a6c0 0x7fa3741efa30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740cd070 0x7fa37414ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 --2- 172.21.15.6:0/1855623930 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3740cca50 0x7fa3740c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:00.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 >> 172.21.15.6:0/1855623930 conn(0x7fa3740b23f0 msgr2=0x7fa3740b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:00.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.970+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 shutdown_connections 2024-03-18T14:42:00.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:00.972+0000 7fa37359e640 1 -- 172.21.15.6:0/1855623930 wait complete. 2024-03-18T14:42:00.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.5 2024-03-18T14:42:00.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.0 2024-03-18T14:42:00.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.4 2024-03-18T14:42:00.982 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.7 2024-03-18T14:42:00.983 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2024-03-18T14:42:00.986 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.6 2024-03-18T14:42:01.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-18T14:42:01.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-18T14:42:01.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-18T14:42:01.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-18T14:42:01.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-18T14:42:01.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-18T14:42:01.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-18T14:42:01.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-18T14:42:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.126+0000 7fb3e46ac640 1 Processor -- start 2024-03-18T14:42:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.126+0000 7fb3e46ac640 1 -- start start 2024-03-18T14:42:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.127+0000 7fb3e46ac640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb3dc06bcb0 0x7fb3dc06de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.127+0000 7fb3e46ac640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc0aec90 0x7fb3dc154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.127+0000 7fb3e46ac640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3dc06e340 con 0x7fb3dc0af0f0 2024-03-18T14:42:02.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.127+0000 7fb3e46ac640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb3dc15a360 con 0x7fb3dc06bcb0 2024-03-18T14:42:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.127+0000 7fb3e46ac640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb3dc15e0d0 con 0x7fb3dc0aec90 2024-03-18T14:42:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.128+0000 7fb3e1c47640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc0aec90 0x7fb3dc154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.128+0000 7fb3e2c49640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb3dc0af0f0 0x7fb3dc0cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45832/0 (socket says 172.21.15.6:45832) 2024-03-18T14:42:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.128+0000 7fb3e2c49640 1 -- 172.21.15.6:0/2446429608 learned_addr learned my addr 172.21.15.6:0/2446429608 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.128+0000 7fb3e1c47640 1 -- 172.21.15.6:0/2446429608 >> v2:172.21.15.6:3300/0 conn(0x7fb3dc06bcb0 msgr2=0x7fb3dc06de00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T14:42:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.129+0000 7fb3e2448640 1 --2- 172.21.15.6:0/2446429608 >> v2:172.21.15.6:3300/0 conn(0x7fb3dc06bcb0 0x7fb3dc06de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.129+0000 7fb3e1c47640 1 --2- 172.21.15.6:0/2446429608 >> v2:172.21.15.6:3300/0 conn(0x7fb3dc06bcb0 0x7fb3dc06de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.129+0000 7fb3e1c47640 1 -- 172.21.15.6:0/2446429608 >> v1:172.21.15.92:6789/0 conn(0x7fb3dc0af0f0 legacy=0x7fb3dc0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.129+0000 7fb3e1c47640 1 -- 172.21.15.6:0/2446429608 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3dc15e3b0 con 0x7fb3dc0aec90 2024-03-18T14:42:02.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.130+0000 7fb3e1c47640 1 --2- 172.21.15.6:0/2446429608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc0aec90 0x7fb3dc154170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097a0 tx=0x7fb3d802b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65a45762fa3a9c1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.130+0000 7fb3e2448640 1 --2- 172.21.15.6:0/2446429608 >> v2:172.21.15.6:3300/0 conn(0x7fb3dc06bcb0 0x7fb3dc06de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.131+0000 7fb3e1446640 1 -- 172.21.15.6:0/2446429608 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8037070 con 0x7fb3dc0aec90 2024-03-18T14:42:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.131+0000 7fb3e1446640 1 -- 172.21.15.6:0/2446429608 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d8030710 con 0x7fb3dc0aec90 2024-03-18T14:42:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.132+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2446429608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc0aec90 msgr2=0x7fb3dc154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.132+0000 7fb3e46ac640 1 --2- 172.21.15.6:0/2446429608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc0aec90 0x7fb3dc154170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097a0 tx=0x7fb3d802b7b0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.133+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2446429608 shutdown_connections 2024-03-18T14:42:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.133+0000 7fb3e46ac640 1 --2- 172.21.15.6:0/2446429608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc0aec90 0x7fb3dc154170 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.133+0000 7fb3e46ac640 1 --2- 172.21.15.6:0/2446429608 >> v2:172.21.15.6:3300/0 conn(0x7fb3dc06bcb0 0x7fb3dc06de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.133+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2446429608 >> 172.21.15.6:0/2446429608 conn(0x7fb3dc06b1c0 msgr2=0x7fb3dc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.133+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2446429608 shutdown_connections 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.133+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2446429608 wait complete. 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.135+0000 7fb3e46ac640 1 Processor -- start 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.135+0000 7fb3e46ac640 1 -- start start 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.136+0000 7fb3e46ac640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc06bcb0 0x7fb3dc148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.136+0000 7fb3e46ac640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3dc0aec90 0x7fb3dc149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e1c47640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3dc0aec90 0x7fb3dc149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e1c47640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3dc0aec90 0x7fb3dc149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56624/0 (socket says 172.21.15.6:56624) 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e46ac640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3dc13c4f0 0x7fb3dc139400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e46ac640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb3dc15db20 con 0x7fb3dc0aec90 2024-03-18T14:42:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e46ac640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb3dc15d9a0 con 0x7fb3dc13c4f0 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e46ac640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb3dc1546b0 con 0x7fb3dc06bcb0 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e1c47640 1 -- 172.21.15.6:0/2972738495 learned_addr learned my addr 172.21.15.6:0/2972738495 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.137+0000 7fb3e2c49640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3dc13c4f0 0x7fb3dc139400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.138+0000 7fb3e1c47640 1 -- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc06bcb0 msgr2=0x7fb3dc148f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.138+0000 7fb3e1c47640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc06bcb0 0x7fb3dc148f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.138+0000 7fb3e1c47640 1 -- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3dc13c4f0 msgr2=0x7fb3dc139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.138+0000 7fb3e1c47640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3dc13c4f0 0x7fb3dc139400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.138+0000 7fb3e1c47640 1 -- 172.21.15.6:0/2972738495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3dc15cb50 con 0x7fb3dc0aec90 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.138+0000 7fb3e1c47640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3dc0aec90 0x7fb3dc149440 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802bd40 tx=0x7fb3d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.140+0000 7fb3d2ffd640 1 -- 172.21.15.6:0/2972738495 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8041020 con 0x7fb3dc0aec90 2024-03-18T14:42:02.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.140+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2972738495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3dc13ca00 con 0x7fb3dc0aec90 2024-03-18T14:42:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.140+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2972738495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3dc13ccc0 con 0x7fb3dc0aec90 2024-03-18T14:42:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.143+0000 7fb3d2ffd640 1 -- 172.21.15.6:0/2972738495 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d803d670 con 0x7fb3dc0aec90 2024-03-18T14:42:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.143+0000 7fb3d2ffd640 1 -- 172.21.15.6:0/2972738495 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8037040 con 0x7fb3dc0aec90 2024-03-18T14:42:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.143+0000 7fb3d2ffd640 1 -- 172.21.15.6:0/2972738495 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb3d8044020 con 0x7fb3dc0aec90 2024-03-18T14:42:02.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.143+0000 7fb3d2ffd640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb3c006bcd0 0x7fb3c006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.144+0000 7fb3e2448640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb3c006bcd0 0x7fb3c006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.144+0000 7fb3d2ffd640 1 -- 172.21.15.6:0/2972738495 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb3d80acd20 con 0x7fb3dc0aec90 2024-03-18T14:42:02.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.144+0000 7fb3e2448640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb3c006bcd0 0x7fb3c006e190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006b10 tx=0x7fb3cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.146+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2972738495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3dc04f5b0 con 0x7fb3dc0aec90 2024-03-18T14:42:02.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.155+0000 7fb3d2ffd640 1 -- 172.21.15.6:0/2972738495 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb3d804b030 con 0x7fb3dc0aec90 2024-03-18T14:42:02.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.183+0000 7f85c6bcd640 1 Processor -- start 2024-03-18T14:42:02.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.183+0000 7f85c6bcd640 1 -- start start 2024-03-18T14:42:02.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.184+0000 7f85c6bcd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f85c00b39e0 0x7f85c00cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.184+0000 7f85c6bcd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85c00af140 con 0x7f85c00aed60 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.184+0000 7f85c6bcd640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85c016b970 con 0x7f85c008a5b0 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.184+0000 7f85c6bcd640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f85c006e7c0 con 0x7f85c00b39e0 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.184+0000 7f85c516a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f85c00b39e0 0x7f85c00cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.185+0000 7f85c516a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f85c00b39e0 0x7f85c00cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56632/0 (socket says 172.21.15.6:56632) 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.185+0000 7f85c516a640 1 -- 172.21.15.6:0/1404961191 learned_addr learned my addr 172.21.15.6:0/1404961191 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.185+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883273002 0 0) 0x7f85c016b970 con 0x7f85c008a5b0 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.187+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85a8003500 con 0x7f85c008a5b0 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.187+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1960595237 0 0) 0x7f85c00af140 con 0x7f85c00aed60 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.187+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85c016b970 con 0x7f85c00aed60 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.186+0000 7f0629680640 1 Processor -- start 2024-03-18T14:42:02.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.187+0000 7f0629680640 1 -- start start 2024-03-18T14:42:02.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.187+0000 7fa63087d640 1 Processor -- start 2024-03-18T14:42:02.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.187+0000 7fa63087d640 1 -- start start 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1768328138 0 0) 0x7f85c016b970 con 0x7f85c00aed60 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 >> v2:172.21.15.6:3300/0 conn(0x7f85c00b39e0 msgr2=0x7f85c00cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 --2- 172.21.15.6:0/1404961191 >> v2:172.21.15.6:3300/0 conn(0x7f85c00b39e0 0x7f85c00cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 >> v1:172.21.15.92:6789/0 conn(0x7f85c008a5b0 legacy=0x7f85c00b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c016bc50 con 0x7f85c00aed60 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f85b4001670 con 0x7f85c00aed60 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 >> v1:172.21.15.6:6789/0 conn(0x7f85c00aed60 legacy=0x7f85c006e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a8003cd0 0x7f85a80040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a8004610 0x7f85a800d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f0629680640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f062406fe70 0x7f062406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f0629680640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0624072220 0x7f0624072650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f0629680640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06240c3ab0 con 0x7f06240c36d0 2024-03-18T14:42:02.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f0629680640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f062406b700 con 0x7f0624072220 2024-03-18T14:42:02.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f0629680640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0624072b90 con 0x7f062406fe70 2024-03-18T14:42:02.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85a800d950 0x7f85a800fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85c006e7c0 con 0x7f85a8004610 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85a8010510 con 0x7f85a800d950 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f85a8010710 con 0x7f85a8003cd0 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f85c516a640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85a800d950 0x7f85a800fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f85c4969640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a8003cd0 0x7f85a80040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f85bffff640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a8004610 0x7f85a800d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7fa63087d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa624037680 0x7fa624057680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f0622ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f06240c36d0 0x7f062406e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53180/0 (socket says 172.21.15.6:53180) 2024-03-18T14:42:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.188+0000 7f0622ffd640 1 -- 172.21.15.6:0/4003707304 learned_addr learned my addr 172.21.15.6:0/4003707304 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f85c516a640 1 -- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a8003cd0 msgr2=0x7f85a80040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f85c516a640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a8003cd0 0x7f85a80040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f85c516a640 1 -- 172.21.15.6:0/1404961191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a8004610 msgr2=0x7f85a800d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f85c516a640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85a8004610 0x7f85a800d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f85c516a640 1 -- 172.21.15.6:0/1404961191 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c016b970 con 0x7f85a800d950 2024-03-18T14:42:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7fa63087d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa624052480 0x7fa6240528b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7fa62ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa624037680 0x7fa624057680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7fa62ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa624037680 0x7fa624057680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56648/0 (socket says 172.21.15.6:56648) 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7fa62a59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa624037a60 0x7fa6240366a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53192/0 (socket says 172.21.15.6:53192) 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7fa62ad9d640 1 -- 172.21.15.6:0/1391613629 learned_addr learned my addr 172.21.15.6:0/1391613629 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7fa62b59e640 1 --2- 172.21.15.6:0/1391613629 >> v2:172.21.15.92:3300/0 conn(0x7fa624052480 0x7fa6240528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f06227fc640 1 --2- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f062406fe70 0x7f062406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f06237fe640 1 --2- 172.21.15.6:0/4003707304 >> v2:172.21.15.92:3300/0 conn(0x7f0624072220 0x7f0624072650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f0621ffb640 1 -- 172.21.15.6:0/4003707304 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777304848 0 0) 0x7f06240c3ab0 con 0x7f06240c36d0 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f0621ffb640 1 -- 172.21.15.6:0/4003707304 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f060c003520 con 0x7f06240c36d0 2024-03-18T14:42:02.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7ff4b099d640 1 Processor -- start 2024-03-18T14:42:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f06237fe640 1 -- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f062406fe70 msgr2=0x7f062406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f06237fe640 1 --2- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f062406fe70 0x7f062406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f06237fe640 1 -- 172.21.15.6:0/4003707304 >> v1:172.21.15.6:6789/0 conn(0x7f06240c36d0 legacy=0x7f062406e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f06237fe640 1 -- 172.21.15.6:0/4003707304 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0624166ba0 con 0x7f0624072220 2024-03-18T14:42:02.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f1bf3d32640 1 Processor -- start 2024-03-18T14:42:02.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.189+0000 7f49db315640 1 Processor -- start 2024-03-18T14:42:02.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7f49db315640 1 -- start start 2024-03-18T14:42:02.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa624038200 con 0x7fa624037a60 2024-03-18T14:42:02.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7f1bf3d32640 1 -- start start 2024-03-18T14:42:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa6240558f0 con 0x7fa624037680 2024-03-18T14:42:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7f85c516a640 1 -- 172.21.15.6:0/1404961191 reap_dead start 2024-03-18T14:42:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7fd7c12f7640 1 Processor -- start 2024-03-18T14:42:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7fd7c12f7640 1 -- start start 2024-03-18T14:42:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7ff4b099d640 1 -- start start 2024-03-18T14:42:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7fa629d9b640 1 -- 172.21.15.6:0/1391613629 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077039091 0 0) 0x7fa620001110 con 0x7fa624037a60 2024-03-18T14:42:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa624055bd0 con 0x7fa624052480 2024-03-18T14:42:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7fa629d9b640 1 -- 172.21.15.6:0/1391613629 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa614003520 con 0x7fa624037a60 2024-03-18T14:42:02.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7f85c516a640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85a800d950 0x7f85a800fd40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f85b8007730 tx=0x7f85a8004d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca84dcba875e9da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.190+0000 7f06237fe640 1 --2- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f0624072650 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f061c009d90 tx=0x7f061c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c30733f90fb7f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf3d32640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1bec15bd50 0x7f1bec0b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf3d32640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1bec0d14e0 0x7f1bec0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f0621ffb640 1 -- 172.21.15.6:0/4003707304 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061c039070 con 0x7f0624072220 2024-03-18T14:42:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f0621ffb640 1 -- 172.21.15.6:0/4003707304 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061c02cc60 con 0x7f0624072220 2024-03-18T14:42:02.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f0621ffb640 1 -- 172.21.15.6:0/4003707304 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061c034660 con 0x7f0624072220 2024-03-18T14:42:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf3d32640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec0b0ae0 0x7f1bec15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf3d32640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1bec0b12f0 con 0x7f1bec15bd50 2024-03-18T14:42:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf3d32640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1bec06b940 con 0x7f1bec0d14e0 2024-03-18T14:42:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf3d32640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec15c130 con 0x7f1bec0b0ae0 2024-03-18T14:42:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f49db315640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f49cc0e3260 0x7f49cc04f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf1ace640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1bec15bd50 0x7f1bec0b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf1ace640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1bec15bd50 0x7f1bec0b18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56656/0 (socket says 172.21.15.6:56656) 2024-03-18T14:42:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf1ace640 1 -- 172.21.15.6:0/2947248987 learned_addr learned my addr 172.21.15.6:0/2947248987 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf22cf640 1 --2- 172.21.15.6:0/2947248987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec0b0ae0 0x7f1bec15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fd7c12f7640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0afa60 0x7fd7bc0d1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fd7c12f7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc0cc940 con 0x7fd7bc15af00 2024-03-18T14:42:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fd7c12f7640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc0cc380 con 0x7fd7bc0cf720 2024-03-18T14:42:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fd7c12f7640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc15b2e0 con 0x7fd7bc0afa60 2024-03-18T14:42:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f49db315640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49cc0e3640 con 0x7f49cc04fdb0 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fd7ba7fc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0afa60 0x7fd7bc0d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fd7ba7fc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0afa60 0x7fd7bc0d1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50946/0 (socket says 172.21.15.6:50946) 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fd7ba7fc640 1 -- 172.21.15.6:0/2049438324 learned_addr learned my addr 172.21.15.6:0/2049438324 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7ff4b099d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7ff4b099d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4ac06b1c0 0x7ff4ac06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7ff4b099d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4ac0700d0 0x7ff4ac070560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7ff4b099d640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff4ac06bb40 con 0x7ff4ac0700d0 2024-03-18T14:42:02.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7ff4b099d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff4ac0cd4a0 con 0x7ff4ac06b1c0 2024-03-18T14:42:02.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fa62ad9d640 1 -- 172.21.15.6:0/1391613629 >> v2:172.21.15.92:3300/0 conn(0x7fa624052480 msgr2=0x7fa6240528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fa62ad9d640 1 --2- 172.21.15.6:0/1391613629 >> v2:172.21.15.92:3300/0 conn(0x7fa624052480 0x7fa6240528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fa62ad9d640 1 -- 172.21.15.6:0/1391613629 >> v1:172.21.15.6:6789/0 conn(0x7fa624037a60 legacy=0x7fa6240366a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7fa62ad9d640 1 -- 172.21.15.6:0/1391613629 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa624051c80 con 0x7fa624037680 2024-03-18T14:42:02.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7ff4b099d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff4ac0cd780 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f49db315640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49cc053930 con 0x7f49cc053380 2024-03-18T14:42:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f49db315640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f49cc053c10 con 0x7f49cc0e3260 2024-03-18T14:42:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7fd7ba7fc640 1 -- 172.21.15.6:0/2049438324 >> v1:172.21.15.92:6789/0 conn(0x7fd7bc0cf720 legacy=0x7fd7bc0adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7fd7ba7fc640 1 -- 172.21.15.6:0/2049438324 >> v1:172.21.15.6:6789/0 conn(0x7fd7bc15af00 legacy=0x7fd7bc0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7fd7ba7fc640 1 -- 172.21.15.6:0/2049438324 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7bc0cc660 con 0x7fd7bc0afa60 2024-03-18T14:42:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f49dab14640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f49cc053380 0x7f49cc037bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45866/0 (socket says 172.21.15.6:45866) 2024-03-18T14:42:02.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f49dab14640 1 -- 172.21.15.6:0/3650989482 learned_addr learned my addr 172.21.15.6:0/3650989482 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f0629680640 1 -- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 msgr2=0x7f0624072650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f0629680640 1 --2- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f0624072650 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f061c009d90 tx=0x7f061c02beb0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f0629680640 1 -- 172.21.15.6:0/4003707304 shutdown_connections 2024-03-18T14:42:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f0629680640 1 --2- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f0624072650 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f0629680640 1 --2- 172.21.15.6:0/4003707304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f062406fe70 0x7f062406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f0629680640 1 -- 172.21.15.6:0/4003707304 >> 172.21.15.6:0/4003707304 conn(0x7f06240d0150 msgr2=0x7f06240d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7fa62ad9d640 1 --2- 172.21.15.6:0/1391613629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037680 0x7fa624057680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa6240204d0 tx=0x7fa61c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba94d64f98b3afb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f49da313640 1 --2- 172.21.15.6:0/3650989482 >> v2:172.21.15.6:3300/0 conn(0x7f49cc0e3260 0x7f49cc04f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7ff4aaffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4ac06b1c0 0x7ff4ac06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7ff4aaffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4ac06b1c0 0x7ff4ac06b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40226/0 (socket says 172.21.15.6:40226) 2024-03-18T14:42:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7ff4aaffd640 1 -- 172.21.15.6:0/1545857666 learned_addr learned my addr 172.21.15.6:0/1545857666 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7ff4aaffd640 1 -- 172.21.15.6:0/1545857666 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 msgr2=0x7ff4ac0b6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7ff4aaffd640 1 --2- 172.21.15.6:0/1545857666 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7ff4aaffd640 1 -- 172.21.15.6:0/1545857666 >> v2:172.21.15.6:3300/0 conn(0x7ff4ac0700d0 msgr2=0x7ff4ac070560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T14:42:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8009d00 con 0x7f85a800d950 2024-03-18T14:42:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b802ccc0 con 0x7f85a800d950 2024-03-18T14:42:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 901464642 0 0) 0x7f49cc053930 con 0x7f49cc053380 2024-03-18T14:42:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49c0003520 con 0x7f49cc053380 2024-03-18T14:42:02.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f85bf7fe640 1 -- 172.21.15.6:0/1404961191 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8003b90 con 0x7f85a800d950 2024-03-18T14:42:02.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf12cd640 1 --2- 172.21.15.6:0/2947248987 >> v2:172.21.15.92:3300/0 conn(0x7f1bec0d14e0 0x7f1bec0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf1ace640 1 -- 172.21.15.6:0/2947248987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec0b0ae0 msgr2=0x7f1bec15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.191+0000 7f1bf1ace640 1 --2- 172.21.15.6:0/2947248987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec0b0ae0 0x7f1bec15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f1bf1ace640 1 -- 172.21.15.6:0/2947248987 >> v2:172.21.15.92:3300/0 conn(0x7f1bec0d14e0 msgr2=0x7f1bec0b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f1bf1ace640 1 --2- 172.21.15.6:0/2947248987 >> v2:172.21.15.92:3300/0 conn(0x7f1bec0d14e0 0x7f1bec0b1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.192+0000 7f1bf1ace640 1 -- 172.21.15.6:0/2947248987 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec0b1010 con 0x7f1bec15bd50 2024-03-18T14:42:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7ff4abfff640 1 --2- 172.21.15.6:0/1545857666 >> v2:172.21.15.6:3300/0 conn(0x7ff4ac0700d0 0x7ff4ac070560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7ff4aaffd640 1 --2- 172.21.15.6:0/1545857666 >> v2:172.21.15.6:3300/0 conn(0x7ff4ac0700d0 0x7ff4ac070560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f0629680640 1 -- 172.21.15.6:0/4003707304 shutdown_connections 2024-03-18T14:42:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f0629680640 1 -- 172.21.15.6:0/4003707304 wait complete. 2024-03-18T14:42:02.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1479850856 0 0) 0x7f49c0003520 con 0x7f49cc053380 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 >> v2:172.21.15.6:3300/0 conn(0x7f49cc0e3260 msgr2=0x7f49cc04f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 --2- 172.21.15.6:0/3650989482 >> v2:172.21.15.6:3300/0 conn(0x7f49cc0e3260 0x7f49cc04f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 >> v1:172.21.15.6:6789/0 conn(0x7f49cc04fdb0 legacy=0x7f49cc052c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49cc053ef0 con 0x7f49cc053380 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f49d0001690 con 0x7f49cc053380 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 >> v1:172.21.15.92:6789/0 conn(0x7f49cc053380 legacy=0x7f49cc037bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49c00039b0 0x7f49c0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49c0004800 0x7f49c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00051c0 0x7f49c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc053c10 con 0x7f49c0004800 2024-03-18T14:42:02.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85a800d950 msgr2=0x7f85a800fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49c000c040 con 0x7f49c00051c0 2024-03-18T14:42:02.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f49c000c240 con 0x7f49c00039b0 2024-03-18T14:42:02.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7fd7ba7fc640 1 --2- 172.21.15.6:0/2049438324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0afa60 0x7fd7bc0d1300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400bf20 tx=0x7fd7b4030360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433ab24c7835fb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85a800d950 0x7f85a800fd40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f85b8007730 tx=0x7f85a8004d50 comp rx=0 tx=0).stop 2024-03-18T14:42:02.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7ff4aaffd640 1 -- 172.21.15.6:0/1545857666 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ac0cda60 con 0x7ff4ac06b1c0 2024-03-18T14:42:02.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7ff4abfff640 1 --2- 172.21.15.6:0/1545857666 >> v2:172.21.15.6:3300/0 conn(0x7ff4ac0700d0 0x7ff4ac070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:02.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7fd7b9ffb640 1 -- 172.21.15.6:0/2049438324 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400b2d0 con 0x7fd7bc0afa60 2024-03-18T14:42:02.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7fd7b9ffb640 1 -- 172.21.15.6:0/2049438324 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b400b470 con 0x7fd7bc0afa60 2024-03-18T14:42:02.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49dab14640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00051c0 0x7f49c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49da313640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49c00039b0 0x7f49c0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1404961191 shutdown_connections 2024-03-18T14:42:02.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1404961191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85a800d950 0x7f85a800fd40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1404961191 >> 172.21.15.6:0/1404961191 conn(0x7f85c006b1c0 msgr2=0x7f85c006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f49d9b12640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49c0004800 0x7f49c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1404961191 shutdown_connections 2024-03-18T14:42:02.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7fa629d9b640 1 -- 172.21.15.6:0/1391613629 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0317c0 con 0x7fa624037680 2024-03-18T14:42:02.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7fa629d9b640 1 -- 172.21.15.6:0/1391613629 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61c037070 con 0x7fa624037680 2024-03-18T14:42:02.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7fa629d9b640 1 -- 172.21.15.6:0/1391613629 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c031b60 con 0x7fa624037680 2024-03-18T14:42:02.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49da313640 1 -- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00051c0 msgr2=0x7f49c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49da313640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00051c0 0x7f49c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49da313640 1 -- 172.21.15.6:0/3650989482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49c0004800 msgr2=0x7f49c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49da313640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49c0004800 0x7f49c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49da313640 1 -- 172.21.15.6:0/3650989482 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49cc053ef0 con 0x7f49c00039b0 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49dab14640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00051c0 0x7f49c000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49dab14640 1 -- 172.21.15.6:0/3650989482 reap_dead start 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f49d9b12640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49c0004800 0x7f49c0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7ff4aaffd640 1 --2- 172.21.15.6:0/1545857666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 0x7ff4ac06b600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0ce320 tx=0x7ff4a402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8890a659fef27ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7ff4aa7fc640 1 -- 172.21.15.6:0/1545857666 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a402cb00 con 0x7ff4ac06b1c0 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7f1bf1ace640 1 --2- 172.21.15.6:0/2947248987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec15bd50 0x7f1bec0b18e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0cb850 tx=0x7f1be802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c567ccbeadca058b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f1bf0acc640 1 -- 172.21.15.6:0/2947248987 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be802e070 con 0x7f1bec15bd50 2024-03-18T14:42:02.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f1bf0acc640 1 -- 172.21.15.6:0/2947248987 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be8037070 con 0x7f1bec15bd50 2024-03-18T14:42:02.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f1bf0acc640 1 -- 172.21.15.6:0/2947248987 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be802e070 con 0x7f1bec15bd50 2024-03-18T14:42:02.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037680 msgr2=0x7fa624057680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7fa63087d640 1 --2- 172.21.15.6:0/1391613629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037680 0x7fa624057680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa6240204d0 tx=0x7fa61c02b860 comp rx=0 tx=0).stop 2024-03-18T14:42:02.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.193+0000 7fd7b9ffb640 1 -- 172.21.15.6:0/2049438324 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4009840 con 0x7fd7bc0afa60 2024-03-18T14:42:02.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f0629680640 1 Processor -- start 2024-03-18T14:42:02.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1404961191 wait complete. 2024-03-18T14:42:02.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f0629680640 1 -- start start 2024-03-18T14:42:02.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7ff4aa7fc640 1 -- 172.21.15.6:0/1545857666 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a402cca0 con 0x7ff4ac06b1c0 2024-03-18T14:42:02.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.194+0000 7ff4aa7fc640 1 -- 172.21.15.6:0/1545857666 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a402cb00 con 0x7ff4ac06b1c0 2024-03-18T14:42:02.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2947248987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec15bd50 msgr2=0x7f1bec0b18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2947248987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec15bd50 0x7f1bec0b18e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0cb850 tx=0x7f1be802b880 comp rx=0 tx=0).stop 2024-03-18T14:42:02.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2947248987 shutdown_connections 2024-03-18T14:42:02.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 shutdown_connections 2024-03-18T14:42:02.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2947248987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec0b0ae0 0x7f1bec15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2947248987 >> v2:172.21.15.92:3300/0 conn(0x7f1bec0d14e0 0x7f1bec0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2947248987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec15bd50 0x7f1bec0b18e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2947248987 >> 172.21.15.6:0/2947248987 conn(0x7f1bec099a50 msgr2=0x7f1bec0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7fa63087d640 1 --2- 172.21.15.6:0/1391613629 >> v2:172.21.15.92:3300/0 conn(0x7fa624052480 0x7fa6240528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7ff4b099d640 1 -- 172.21.15.6:0/1545857666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 msgr2=0x7ff4ac06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7ff4b099d640 1 --2- 172.21.15.6:0/1545857666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 0x7ff4ac06b600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0ce320 tx=0x7ff4a402bf60 comp rx=0 tx=0).stop 2024-03-18T14:42:02.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7fa63087d640 1 --2- 172.21.15.6:0/1391613629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037680 0x7fa624057680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f49da313640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49c00039b0 0x7f49c0003db0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009ac0 tx=0x7f49c0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1707391a5a4f6a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 >> 172.21.15.6:0/1391613629 conn(0x7fa624020030 msgr2=0x7fa624056d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7ff4b099d640 1 -- 172.21.15.6:0/1545857666 shutdown_connections 2024-03-18T14:42:02.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7ff4b099d640 1 --2- 172.21.15.6:0/1545857666 >> v2:172.21.15.6:3300/0 conn(0x7ff4ac0700d0 0x7ff4ac070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7ff4b099d640 1 --2- 172.21.15.6:0/1545857666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 0x7ff4ac06b600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2947248987 shutdown_connections 2024-03-18T14:42:02.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7ff4b099d640 1 --2- 172.21.15.6:0/1545857666 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7ff4b099d640 1 -- 172.21.15.6:0/1545857666 >> 172.21.15.6:0/1545857666 conn(0x7ff4ac0b23f0 msgr2=0x7ff4ac0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2947248987 wait complete. 2024-03-18T14:42:02.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 shutdown_connections 2024-03-18T14:42:02.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4039070 con 0x7f49c00039b0 2024-03-18T14:42:02.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c402c660 con 0x7f49c00039b0 2024-03-18T14:42:02.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f49d9311640 1 -- 172.21.15.6:0/3650989482 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4009690 con 0x7f49c00039b0 2024-03-18T14:42:02.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7ff4b099d640 1 -- 172.21.15.6:0/1545857666 shutdown_connections 2024-03-18T14:42:02.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7fd7c12f7640 1 -- 172.21.15.6:0/2049438324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0afa60 msgr2=0x7fd7bc0d1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/2049438324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0afa60 0x7fd7bc0d1300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400bf20 tx=0x7fd7b4030360 comp rx=0 tx=0).stop 2024-03-18T14:42:02.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7ff4b099d640 1 -- 172.21.15.6:0/1545857666 wait complete. 2024-03-18T14:42:02.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7fd7c12f7640 1 -- 172.21.15.6:0/2049438324 shutdown_connections 2024-03-18T14:42:02.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/2049438324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0afa60 0x7fd7bc0d1300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7fd7c12f7640 1 -- 172.21.15.6:0/2049438324 >> 172.21.15.6:0/2049438324 conn(0x7fd7bc099a50 msgr2=0x7fd7bc0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7fa63087d640 1 -- 172.21.15.6:0/1391613629 wait complete. 2024-03-18T14:42:02.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f49db315640 1 -- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49c00039b0 msgr2=0x7f49c0003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f49db315640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49c00039b0 0x7f49c0003db0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009ac0 tx=0x7f49c0005830 comp rx=0 tx=0).stop 2024-03-18T14:42:02.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f49db315640 1 -- 172.21.15.6:0/3650989482 shutdown_connections 2024-03-18T14:42:02.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f49db315640 1 --2- 172.21.15.6:0/3650989482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49c00039b0 0x7f49c0003db0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009ac0 tx=0x7f49c0005830 comp rx=0 tx=0).stop 2024-03-18T14:42:02.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f49db315640 1 -- 172.21.15.6:0/3650989482 >> 172.21.15.6:0/3650989482 conn(0x7f49cc009fb0 msgr2=0x7f49cc00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.195+0000 7f85c6bcd640 1 Processor -- start 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f85c6bcd640 1 -- start start 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85c6bcd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a800d950 0x7f85c01625f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85c6bcd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85c00ccf70 0x7f85c00cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85c6bcd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85c00cf560 0x7f85c00cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f0629680640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f06240c3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f0629680640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06240c36d0 0x7f06240c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f0629680640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06241f6130 0x7f062415ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f06237fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06241f6130 0x7f062415ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f06237fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06241f6130 0x7f062415ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56696/0 (socket says 172.21.15.6:56696) 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f06237fe640 1 -- 172.21.15.6:0/743596240 learned_addr learned my addr 172.21.15.6:0/743596240 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f062406cc40 con 0x7f06241f6130 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f062406c960 con 0x7f0624072220 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f06241631b0 con 0x7f06240c36d0 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.196+0000 7f0622ffd640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f06240c3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f0622ffd640 1 -- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06240c36d0 msgr2=0x7f06240c61c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85c6bcd640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85c00cc000 con 0x7f85c00cf560 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85c6bcd640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85c00b2d60 con 0x7f85c00ccf70 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85c6bcd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f85c00cc180 con 0x7f85a800d950 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85bffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85c00ccf70 0x7f85c00cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85bffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85c00ccf70 0x7f85c00cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40250/0 (socket says 172.21.15.6:40250) 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85bffff640 1 -- 172.21.15.6:0/1159515373 learned_addr learned my addr 172.21.15.6:0/1159515373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f0622ffd640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06240c36d0 0x7f06240c61c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f0622ffd640 1 -- 172.21.15.6:0/743596240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06241f6130 msgr2=0x7f062415ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f0622ffd640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06241f6130 0x7f062415ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f0622ffd640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06241f4740 con 0x7f0624072220 2024-03-18T14:42:02.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7f85c516a640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85c00cf560 0x7f85c00cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f1bf3d32640 1 Processor -- start 2024-03-18T14:42:02.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f1bf3d32640 1 -- start start 2024-03-18T14:42:02.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85bffff640 1 -- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a800d950 msgr2=0x7f85c01625f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T14:42:02.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85bffff640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a800d950 0x7f85c01625f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85bffff640 1 -- 172.21.15.6:0/1159515373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85c00cf560 msgr2=0x7f85c00cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85bffff640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85c00cf560 0x7f85c00cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85bffff640 1 -- 172.21.15.6:0/1159515373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c00cff50 con 0x7f85c00ccf70 2024-03-18T14:42:02.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f0622ffd640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f06240c3dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f061400c3c0 tx=0x7f0614001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7fd7c12f7640 1 -- 172.21.15.6:0/2049438324 shutdown_connections 2024-03-18T14:42:02.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85bffff640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85c00ccf70 0x7f85c00cd3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85b000e7b0 tx=0x7f85b000ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f06237fe640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06241f6130 0x7f062415ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:02.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85bd7fa640 1 -- 172.21.15.6:0/1159515373 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0001d10 con 0x7f85c00ccf70 2024-03-18T14:42:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f060b7fe640 1 -- 172.21.15.6:0/743596240 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614017070 con 0x7f0624072220 2024-03-18T14:42:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f060b7fe640 1 -- 172.21.15.6:0/743596240 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0614005ce0 con 0x7f0624072220 2024-03-18T14:42:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f060b7fe640 1 -- 172.21.15.6:0/743596240 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06140072d0 con 0x7f0624072220 2024-03-18T14:42:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06240c6960 con 0x7f0624072220 2024-03-18T14:42:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06240c6f20 con 0x7f0624072220 2024-03-18T14:42:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf3d32640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bec0b0ae0 0x7f1bec0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.197+0000 7ff4b099d640 1 Processor -- start 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7ff4b099d640 1 -- start start 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4b099d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 0x7ff4ac0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4b099d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4ac0700d0 0x7ff4ac2005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4b099d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4b099d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac0cd190 con 0x7ff4ac0700d0 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4b099d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac0ce020 con 0x7ff4ac06b1c0 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4b099d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff4ac15bfb0 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf3d32640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec0d14e0 0x7f1bec0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7fa63087d640 1 Processor -- start 2024-03-18T14:42:02.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf3d32640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec15bd50 0x7f1bec15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4abfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4abfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50996/0 (socket says 172.21.15.6:50996) 2024-03-18T14:42:02.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7ff4abfff640 1 -- 172.21.15.6:0/2688563905 learned_addr learned my addr 172.21.15.6:0/2688563905 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf3d32640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec099ed0 con 0x7f1bec0d14e0 2024-03-18T14:42:02.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf3d32640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec06b5d0 con 0x7f1bec0b0ae0 2024-03-18T14:42:02.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf3d32640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec09a050 con 0x7f1bec15bd50 2024-03-18T14:42:02.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf12cd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec0d14e0 0x7f1bec0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf12cd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec0d14e0 0x7f1bec0b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56722/0 (socket says 172.21.15.6:56722) 2024-03-18T14:42:02.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f1bf12cd640 1 -- 172.21.15.6:0/2778348943 learned_addr learned my addr 172.21.15.6:0/2778348943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f49db315640 1 -- 172.21.15.6:0/3650989482 shutdown_connections 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7f1bf1ace640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bec0b0ae0 0x7f1bec0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7f1bf12cd640 1 -- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec15bd50 msgr2=0x7f1bec15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7f1bf12cd640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec15bd50 0x7f1bec15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7f1bf12cd640 1 -- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bec0b0ae0 msgr2=0x7f1bec0c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7f1bf12cd640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bec0b0ae0 0x7f1bec0c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7f1bf12cd640 1 -- 172.21.15.6:0/2778348943 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec0c6990 con 0x7f1bec0d14e0 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c00cd9a0 con 0x7f85c00ccf70 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c00cdf60 con 0x7f85c00ccf70 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f85bd7fa640 1 -- 172.21.15.6:0/1159515373 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b0018070 con 0x7f85c00ccf70 2024-03-18T14:42:02.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.199+0000 7f85bd7fa640 1 -- 172.21.15.6:0/1159515373 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0005330 con 0x7f85c00ccf70 2024-03-18T14:42:02.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7fa63087d640 1 -- start start 2024-03-18T14:42:02.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7f060b7fe640 1 -- 172.21.15.6:0/743596240 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f061401a070 con 0x7f0624072220 2024-03-18T14:42:02.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7f060b7fe640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f061006bca0 0x7f061006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7f060b7fe640 1 -- 172.21.15.6:0/743596240 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f061408a080 con 0x7f0624072220 2024-03-18T14:42:02.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7ff4abfff640 1 -- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 msgr2=0x7ff4ac0ca180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T14:42:02.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7ff4abfff640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 0x7ff4ac0ca180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7ff4abfff640 1 -- 172.21.15.6:0/2688563905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4ac0700d0 msgr2=0x7ff4ac2005f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7ff4abfff640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4ac0700d0 0x7ff4ac2005f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7ff4abfff640 1 -- 172.21.15.6:0/2688563905 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ac0a8a30 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7ff4abfff640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b1180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00c7f0 tx=0x7ff49c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7ff493fff640 1 -- 172.21.15.6:0/2688563905 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c015020 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7ff493fff640 1 -- 172.21.15.6:0/2688563905 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff49c004230 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7ff493fff640 1 -- 172.21.15.6:0/2688563905 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c007620 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.198+0000 7fd7c12f7640 1 -- 172.21.15.6:0/2049438324 wait complete. 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7fd7c12f7640 1 Processor -- start 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7fd7c12f7640 1 -- start start 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7fd7c12f7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7bc0afa60 0x7fd7bc200a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7fd7c12f7640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0cf720 0x7fd7bc0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7fd7c12f7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7bc15af00 0x7fd7bc1fded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7fd7c12f7640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc0cc020 con 0x7fd7bc0afa60 2024-03-18T14:42:02.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7fd7c12f7640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc0adaa0 con 0x7fd7bc15af00 2024-03-18T14:42:02.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7fd7c12f7640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc0ae6d0 con 0x7fd7bc0cf720 2024-03-18T14:42:02.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.200+0000 7f49db315640 1 -- 172.21.15.6:0/3650989482 wait complete. 2024-03-18T14:42:02.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7f49db315640 1 Processor -- start 2024-03-18T14:42:02.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7f49db315640 1 -- start start 2024-03-18T14:42:02.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7fd7baffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7bc0afa60 0x7fd7bc200a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.201+0000 7f85bd7fa640 1 -- 172.21.15.6:0/1159515373 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f85b000c6c0 con 0x7f85c00ccf70 2024-03-18T14:42:02.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7f85bd7fa640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f85ac06bcd0 0x7f85ac06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7fd7baffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7bc0afa60 0x7fd7bc200a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56730/0 (socket says 172.21.15.6:56730) 2024-03-18T14:42:02.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7f85c4969640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f85ac06bcd0 0x7f85ac06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7fd7baffd640 1 -- 172.21.15.6:0/3990138290 learned_addr learned my addr 172.21.15.6:0/3990138290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ac0a8d10 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ac0a92d0 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a0005290 con 0x7f85c00ccf70 2024-03-18T14:42:02.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7f1bf12cd640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec0d14e0 0x7f1bec0b6020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c820 tx=0x7f1bdc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7fa63087d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa624037680 0x7fa624176200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7fa63087d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037a60 0x7fa62404c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7f85bd7fa640 1 -- 172.21.15.6:0/1159515373 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f85b008a890 con 0x7f85c00ccf70 2024-03-18T14:42:02.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.202+0000 7f1be27fc640 1 -- 172.21.15.6:0/2778348943 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc015070 con 0x7f1bec0d14e0 2024-03-18T14:42:02.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0604005290 con 0x7f0624072220 2024-03-18T14:42:02.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49db315640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49cc0e50a0 0x7f49cc04cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49db315640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49cc04d4f0 0x7f49cc0ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49db315640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49cc0e5480 0x7f49cc0cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f1be27fc640 1 -- 172.21.15.6:0/2778348943 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bdc004220 con 0x7f1bec0d14e0 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49db315640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc04f870 con 0x7f49cc0e50a0 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49db315640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc0544b0 con 0x7f49cc04d4f0 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49db315640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f49cc04f9f0 con 0x7f49cc0e5480 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49cc04d4f0 0x7f49cc0ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49da313640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49cc0e50a0 0x7f49cc04cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49cc04d4f0 0x7f49cc0ceb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40286/0 (socket says 172.21.15.6:40286) 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 -- 172.21.15.6:0/2885478297 learned_addr learned my addr 172.21.15.6:0/2885478297 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49dab14640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49cc0e5480 0x7f49cc0cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 -- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49cc0e5480 msgr2=0x7f49cc0cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49cc0e5480 0x7f49cc0cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 -- 172.21.15.6:0/2885478297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49cc0e50a0 msgr2=0x7f49cc04cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49cc0e50a0 0x7f49cc04cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f49d9b12640 1 -- 172.21.15.6:0/2885478297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49cc04ea30 con 0x7f49cc04d4f0 2024-03-18T14:42:02.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f85c4969640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f85ac06bcd0 0x7f85ac06e190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f85b4007a70 tx=0x7f85b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f1be27fc640 1 -- 172.21.15.6:0/2778348943 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc00ce10 con 0x7f1bec0d14e0 2024-03-18T14:42:02.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bec0c6c70 con 0x7f1bec0d14e0 2024-03-18T14:42:02.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bec0c7230 con 0x7f1bec0d14e0 2024-03-18T14:42:02.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7f06227fc640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f061006bca0 0x7f061006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7f49d9b12640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49cc04d4f0 0x7f49cc0ceb60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f49d40500d0 tx=0x7f49d4068c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4ac15e190 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7ff493fff640 1 -- 172.21.15.6:0/2688563905 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff49c018020 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7fa63087d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa624052480 0x7fa624035380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa63087d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa624057bc0 con 0x7fa624037a60 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62a59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037a60 0x7fa62404c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa63087d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa624051970 con 0x7fa624052480 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa63087d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa624036db0 con 0x7fa624037680 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62a59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037a60 0x7fa62404c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56740/0 (socket says 172.21.15.6:56740) 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62ad9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa624037680 0x7fa624176200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62a59c640 1 -- 172.21.15.6:0/2051901770 learned_addr learned my addr 172.21.15.6:0/2051901770 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62b59e640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa624052480 0x7fa624035380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7ff493fff640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff49806bc20 0x7ff49806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62ad9d640 1 -- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa624052480 msgr2=0x7fa624035380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62ad9d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa624052480 0x7fa624035380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62ad9d640 1 -- 172.21.15.6:0/2051901770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037a60 msgr2=0x7fa62404c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62ad9d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037a60 0x7fa62404c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fa62ad9d640 1 -- 172.21.15.6:0/2051901770 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62404b5c0 con 0x7fa624037680 2024-03-18T14:42:02.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7ff493fff640 1 -- 172.21.15.6:0/2688563905 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff49c08ad90 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.205+0000 7ff4ab7fe640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff49806bc20 0x7ff49806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7fd7ba7fc640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0cf720 0x7fd7bc0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7fd7bb7fe640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7bc15af00 0x7fd7bc1fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7fd7ba7fc640 1 -- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7bc15af00 msgr2=0x7fd7bc1fded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7fd7ba7fc640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7bc15af00 0x7fd7bc1fded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.203+0000 7fd7ba7fc640 1 -- 172.21.15.6:0/3990138290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7bc0afa60 msgr2=0x7fd7bc200a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fd7ba7fc640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7bc0afa60 0x7fd7bc200a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fd7ba7fc640 1 -- 172.21.15.6:0/3990138290 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7bc201060 con 0x7fd7bc0cf720 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fd7baffd640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7bc0afa60 0x7fd7bc200a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.204+0000 7fd7ba7fc640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0cf720 0x7fd7bc0b49a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4036000 tx=0x7fd7b4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.205+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bec04f5b0 con 0x7f1bec0d14e0 2024-03-18T14:42:02.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.205+0000 7f06227fc640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f061006bca0 0x7f061006e160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f06240b0300 tx=0x7f0618009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.205+0000 7fa62ad9d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa624037680 0x7fa624176200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa61c033000 tx=0x7fa61c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.205+0000 7ff4ab7fe640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff49806bc20 0x7ff49806e0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00096b0 tx=0x7ff4a0006c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.206+0000 7f1be27fc640 1 -- 172.21.15.6:0/2778348943 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1bdc007830 con 0x7f1bec0d14e0 2024-03-18T14:42:02.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.206+0000 7f1be27fc640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1bd006bc20 0x7f1bd006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.206+0000 7f1be27fc640 1 -- 172.21.15.6:0/2778348943 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1bdc08af00 con 0x7f1bec0d14e0 2024-03-18T14:42:02.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.206+0000 7f1bf1ace640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1bd006bc20 0x7f1bd006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.207+0000 7f1bf1ace640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1bd006bc20 0x7f1bd006e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1be8036000 tx=0x7f1be803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.210+0000 7f1be27fc640 1 -- 172.21.15.6:0/2778348943 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1bdc059140 con 0x7f1bec0d14e0 2024-03-18T14:42:02.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.210+0000 7f85bd7fa640 1 -- 172.21.15.6:0/1159515373 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f85b00590b0 con 0x7f85c00ccf70 2024-03-18T14:42:02.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fd7a37fe640 1 -- 172.21.15.6:0/3990138290 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4004580 con 0x7fd7bc0cf720 2024-03-18T14:42:02.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fa6137fe640 1 -- 172.21.15.6:0/2051901770 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c041070 con 0x7fa624037680 2024-03-18T14:42:02.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fa6137fe640 1 -- 172.21.15.6:0/2051901770 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61c003e50 con 0x7fa624037680 2024-03-18T14:42:02.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fa6137fe640 1 -- 172.21.15.6:0/2051901770 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0317e0 con 0x7fa624037680 2024-03-18T14:42:02.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62404b8a0 con 0x7fa624037680 2024-03-18T14:42:02.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fd7a37fe640 1 -- 172.21.15.6:0/3990138290 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b4004780 con 0x7fd7bc0cf720 2024-03-18T14:42:02.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62404be60 con 0x7fa624037680 2024-03-18T14:42:02.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7ff493fff640 1 -- 172.21.15.6:0/2688563905 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff4ac15e190 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.214+0000 7fd7a37fe640 1 -- 172.21.15.6:0/3990138290 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4009840 con 0x7fd7bc0cf720 2024-03-18T14:42:02.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.215+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7bc201340 con 0x7fd7bc0cf720 2024-03-18T14:42:02.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.215+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7bc201900 con 0x7fd7bc0cf720 2024-03-18T14:42:02.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.216+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa624005700 con 0x7fa624037680 2024-03-18T14:42:02.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.216+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7bc04f5b0 con 0x7fd7bc0cf720 2024-03-18T14:42:02.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.216+0000 7fa6137fe640 1 -- 172.21.15.6:0/2051901770 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa61c04c060 con 0x7fa624037680 2024-03-18T14:42:02.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.217+0000 7fa6137fe640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa61806bca0 0x7fa61806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.217+0000 7fd7a37fe640 1 -- 172.21.15.6:0/3990138290 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd7b4005d10 con 0x7fd7bc0cf720 2024-03-18T14:42:02.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.217+0000 7fa6137fe640 1 -- 172.21.15.6:0/2051901770 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa61c0ac500 con 0x7fa624037680 2024-03-18T14:42:02.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.217+0000 7f49caffd640 1 -- 172.21.15.6:0/2885478297 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4062c30 con 0x7f49cc04d4f0 2024-03-18T14:42:02.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.217+0000 7f49caffd640 1 -- 172.21.15.6:0/2885478297 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d4073040 con 0x7f49cc04d4f0 2024-03-18T14:42:02.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.218+0000 7fa62a59c640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa61806bca0 0x7fa61806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.218+0000 7f060b7fe640 1 -- 172.21.15.6:0/743596240 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f06140581c0 con 0x7f0624072220 2024-03-18T14:42:02.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.217+0000 7fd7a37fe640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd7a806bbd0 0x7fd7a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.218+0000 7fd7a37fe640 1 -- 172.21.15.6:0/3990138290 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd7b40aca60 con 0x7fd7bc0cf720 2024-03-18T14:42:02.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.218+0000 7fd7baffd640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd7a806bbd0 0x7fd7a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.217+0000 7f49caffd640 1 -- 172.21.15.6:0/2885478297 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4073430 con 0x7f49cc04d4f0 2024-03-18T14:42:02.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.218+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49cc04d990 con 0x7f49cc04d4f0 2024-03-18T14:42:02.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.218+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49cc04df50 con 0x7f49cc04d4f0 2024-03-18T14:42:02.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.219+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49cc0052c0 con 0x7f49cc04d4f0 2024-03-18T14:42:02.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.220+0000 7fa62a59c640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa61806bca0 0x7fa61806e160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6240d68e0 tx=0x7fa620007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.220+0000 7fd7baffd640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd7a806bbd0 0x7fd7a806e090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0cd740 tx=0x7fd7b00090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.220+0000 7f49caffd640 1 -- 172.21.15.6:0/2885478297 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f49d4092030 con 0x7f49cc04d4f0 2024-03-18T14:42:02.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.220+0000 7f49caffd640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49b806bc50 0x7f49b806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.221+0000 7f49caffd640 1 -- 172.21.15.6:0/2885478297 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f49d4105a40 con 0x7f49cc04d4f0 2024-03-18T14:42:02.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.221+0000 7f49da313640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49b806bc50 0x7f49b806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.221+0000 7f49da313640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49b806bc50 0x7f49b806e110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f49c4036000 tx=0x7f49c4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.223+0000 7fa6137fe640 1 -- 172.21.15.6:0/2051901770 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa61c07a6d0 con 0x7fa624037680 2024-03-18T14:42:02.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.230+0000 7f49caffd640 1 -- 172.21.15.6:0/2885478297 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f49d40d3c00 con 0x7f49cc04d4f0 2024-03-18T14:42:02.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.232+0000 7fd7a37fe640 1 -- 172.21.15.6:0/3990138290 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd7b407ab20 con 0x7fd7bc0cf720 2024-03-18T14:42:02.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.540+0000 7fb3e46ac640 1 -- 172.21.15.6:0/2972738495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb3dc0b2da0 con 0x7fb3dc0aec90 2024-03-18T14:42:02.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.541+0000 7fb3d2ffd640 1 -- 172.21.15.6:0/2972738495 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb3d8032e20 con 0x7fb3dc0aec90 2024-03-18T14:42:02.543 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.544+0000 7fb3d0ff9640 1 -- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb3c006bcd0 msgr2=0x7fb3c006e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.544+0000 7fb3d0ff9640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb3c006bcd0 0x7fb3c006e190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006b10 tx=0x7fb3cc008040 comp rx=0 tx=0).stop 2024-03-18T14:42:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 -- 172.21.15.6:0/2972738495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3dc0aec90 msgr2=0x7fb3dc149440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3dc0aec90 0x7fb3dc149440 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802bd40 tx=0x7fb3d80046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 -- 172.21.15.6:0/2972738495 shutdown_connections 2024-03-18T14:42:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb3c006bcd0 0x7fb3c006e190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3dc13c4f0 0x7fb3dc139400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3dc0aec90 0x7fb3dc149440 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 --2- 172.21.15.6:0/2972738495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3dc06bcb0 0x7fb3dc148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.545+0000 7fb3d0ff9640 1 -- 172.21.15.6:0/2972738495 >> 172.21.15.6:0/2972738495 conn(0x7fb3dc06b1c0 msgr2=0x7fb3dc0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.546+0000 7fb3d0ff9640 1 -- 172.21.15.6:0/2972738495 shutdown_connections 2024-03-18T14:42:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.546+0000 7fb3d0ff9640 1 -- 172.21.15.6:0/2972738495 wait complete. 2024-03-18T14:42:02.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-03-18T14:42:02.564 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.598+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd7bc0d1940 con 0x7fd7bc0cf720 2024-03-18T14:42:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.599+0000 7fd7a37fe640 1 -- 172.21.15.6:0/3990138290 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd7b407ebc0 con 0x7fd7bc0cf720 2024-03-18T14:42:02.600 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:02.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd7a806bbd0 msgr2=0x7fd7a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd7a806bbd0 0x7fd7a806e090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0cd740 tx=0x7fd7b00090e0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0cf720 msgr2=0x7fd7bc0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0cf720 0x7fd7bc0b49a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4036000 tx=0x7fd7b4004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 shutdown_connections 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd7a806bbd0 0x7fd7a806e090 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7bc15af00 0x7fd7bc1fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7bc0cf720 0x7fd7bc0b49a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 --2- 172.21.15.6:0/3990138290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7bc0afa60 0x7fd7bc200a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 >> 172.21.15.6:0/3990138290 conn(0x7fd7bc099a50 msgr2=0x7fd7bc0ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 shutdown_connections 2024-03-18T14:42:02.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.601+0000 7fd7c12f7640 1 -- 172.21.15.6:0/3990138290 wait complete. 2024-03-18T14:42:02.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.6 2024-03-18T14:42:02.615 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.627+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff4ac04f540 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.627+0000 7ff493fff640 1 -- 172.21.15.6:0/2688563905 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff49c01f020 con 0x7ff4ac15bbd0 2024-03-18T14:42:02.629 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff49806bc20 msgr2=0x7ff49806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff49806bc20 0x7ff49806e0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00096b0 tx=0x7ff4a0006c20 comp rx=0 tx=0).stop 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 msgr2=0x7ff4ac0b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b1180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00c7f0 tx=0x7ff49c00ccb0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 shutdown_connections 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff49806bc20 0x7ff49806e0e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac15bbd0 0x7ff4ac0b1180 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4ac0700d0 0x7ff4ac2005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 --2- 172.21.15.6:0/2688563905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac06b1c0 0x7ff4ac0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 >> 172.21.15.6:0/2688563905 conn(0x7ff4ac0b23f0 msgr2=0x7ff4ac0af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 shutdown_connections 2024-03-18T14:42:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.629+0000 7ff4b099d640 1 -- 172.21.15.6:0/2688563905 wait complete. 2024-03-18T14:42:02.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.632+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1bec0ce180 con 0x7f1bec0d14e0 2024-03-18T14:42:02.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.633+0000 7f1be27fc640 1 -- 172.21.15.6:0/2778348943 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1bdc05d1e0 con 0x7f1bec0d14e0 2024-03-18T14:42:02.635 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:02.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.634+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f85a0005580 con 0x7f85c00ccf70 2024-03-18T14:42:02.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.634+0000 7f85bd7fa640 1 -- 172.21.15.6:0/1159515373 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f85b00589b0 con 0x7f85c00ccf70 2024-03-18T14:42:02.636 INFO:teuthology.orchestra.run.smithi006.stdout:55834574858 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.636+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1bd006bc20 msgr2=0x7f1bd006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.636+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1bd006bc20 0x7f1bd006e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1be8036000 tx=0x7f1be803d040 comp rx=0 tx=0).stop 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.636+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec0d14e0 msgr2=0x7f1bec0b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.636+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec0d14e0 0x7f1bec0b6020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c820 tx=0x7f1bdc00cce0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 shutdown_connections 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1bd006bc20 0x7f1bd006e0e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bec15bd50 0x7f1bec15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1bec0d14e0 0x7f1bec0b6020 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 --2- 172.21.15.6:0/2778348943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bec0b0ae0 0x7f1bec0c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 >> 172.21.15.6:0/2778348943 conn(0x7f1bec099a50 msgr2=0x7f1bec06f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 shutdown_connections 2024-03-18T14:42:02.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.637+0000 7f1bf3d32640 1 -- 172.21.15.6:0/2778348943 wait complete. 2024-03-18T14:42:02.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f85ac06bcd0 msgr2=0x7f85ac06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f85ac06bcd0 0x7f85ac06e190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f85b4007a70 tx=0x7f85b4009040 comp rx=0 tx=0).stop 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85c00ccf70 msgr2=0x7f85c00cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85c00ccf70 0x7f85c00cd3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85b000e7b0 tx=0x7f85b000ec70 comp rx=0 tx=0).stop 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 shutdown_connections 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f85ac06bcd0 0x7f85ac06e190 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85c00cf560 0x7f85c00cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85c00ccf70 0x7f85c00cd3b0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85b000e7b0 tx=0x7f85b000ec70 comp rx=0 tx=0).stop 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 --2- 172.21.15.6:0/1159515373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85a800d950 0x7f85c01625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 >> 172.21.15.6:0/1159515373 conn(0x7f85c006b1c0 msgr2=0x7f85c016e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 shutdown_connections 2024-03-18T14:42:02.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.638+0000 7f85c6bcd640 1 -- 172.21.15.6:0/1159515373 wait complete. 2024-03-18T14:42:02.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.640+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0604005580 con 0x7f0624072220 2024-03-18T14:42:02.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.641+0000 7f060b7fe640 1 -- 172.21.15.6:0/743596240 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f061405c260 con 0x7f0624072220 2024-03-18T14:42:02.642 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:02.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.642+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f49cc0e5ed0 con 0x7f49cc04d4f0 2024-03-18T14:42:02.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.642+0000 7f49caffd640 1 -- 172.21.15.6:0/2885478297 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f49d40d7ca0 con 0x7f49cc04d4f0 2024-03-18T14:42:02.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f061006bca0 msgr2=0x7f061006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f061006bca0 0x7f061006e160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f06240b0300 tx=0x7f0618009110 comp rx=0 tx=0).stop 2024-03-18T14:42:02.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 msgr2=0x7f06240c3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.645 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-03-18T14:42:02.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f06240c3dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f061400c3c0 tx=0x7f0614001e70 comp rx=0 tx=0).stop 2024-03-18T14:42:02.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 shutdown_connections 2024-03-18T14:42:02.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f061006bca0 0x7f061006e160 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06241f6130 0x7f062415ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06240c36d0 0x7f06240c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 --2- 172.21.15.6:0/743596240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0624072220 0x7f06240c3dc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.643+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 >> 172.21.15.6:0/743596240 conn(0x7f06240d0150 msgr2=0x7f06241660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.647 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-03-18T14:42:02.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.644+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 shutdown_connections 2024-03-18T14:42:02.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.644+0000 7f0629680640 1 -- 172.21.15.6:0/743596240 wait complete. 2024-03-18T14:42:02.648 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49b806bc50 msgr2=0x7f49b806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49b806bc50 0x7f49b806e110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f49c4036000 tx=0x7f49c4030000 comp rx=0 tx=0).stop 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49cc04d4f0 msgr2=0x7f49cc0ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49cc04d4f0 0x7f49cc0ceb60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f49d40500d0 tx=0x7f49d4068c80 comp rx=0 tx=0).stop 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 shutdown_connections 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49b806bc50 0x7f49b806e110 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49cc0e5480 0x7f49cc0cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49cc04d4f0 0x7f49cc0ceb60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 --2- 172.21.15.6:0/2885478297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49cc0e50a0 0x7f49cc04cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 >> 172.21.15.6:0/2885478297 conn(0x7f49cc009fb0 msgr2=0x7f49cc0389b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 shutdown_connections 2024-03-18T14:42:02.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.645+0000 7f49db315640 1 -- 172.21.15.6:0/2885478297 wait complete. 2024-03-18T14:42:02.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.646+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa624054ba0 con 0x7fa624037680 2024-03-18T14:42:02.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.647+0000 7fa6137fe640 1 -- 172.21.15.6:0/2051901770 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa61c07e770 con 0x7fa624037680 2024-03-18T14:42:02.650 INFO:teuthology.orchestra.run.smithi006.stdout:55834574860 2024-03-18T14:42:02.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa61806bca0 msgr2=0x7fa61806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa61806bca0 0x7fa61806e160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6240d68e0 tx=0x7fa620007c90 comp rx=0 tx=0).stop 2024-03-18T14:42:02.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa624037680 msgr2=0x7fa624176200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa624037680 0x7fa624176200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa61c033000 tx=0x7fa61c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 shutdown_connections 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa61806bca0 0x7fa61806e160 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa624052480 0x7fa624035380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa624037a60 0x7fa62404c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 --2- 172.21.15.6:0/2051901770 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa624037680 0x7fa624176200 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 >> 172.21.15.6:0/2051901770 conn(0x7fa624020030 msgr2=0x7fa624055e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 shutdown_connections 2024-03-18T14:42:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.650+0000 7fa63087d640 1 -- 172.21.15.6:0/2051901770 wait complete. 2024-03-18T14:42:02.657 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.5 2024-03-18T14:42:02.657 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2024-03-18T14:42:02.658 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.0 2024-03-18T14:42:02.666 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.667 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.7 2024-03-18T14:42:02.667 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-03-18T14:42:02.668 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.668 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:02.668 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-18T14:42:02.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:42:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.806+0000 7f94db26c640 1 Processor -- start 2024-03-18T14:42:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.806+0000 7f94db26c640 1 -- start start 2024-03-18T14:42:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94db26c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d415bd40 0x7f94d40d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94db26c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f94d40d1d90 0x7f94d40cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94db26c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94d40cc870 con 0x7f94d40cfcd0 2024-03-18T14:42:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94db26c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f94d40d0500 con 0x7f94d40d1d90 2024-03-18T14:42:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94db26c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f94d40d07e0 con 0x7f94d415bd40 2024-03-18T14:42:02.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94d9008640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d415bd40 0x7f94d40d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94d8807640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f94d40d1d90 0x7f94d40cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94d9008640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d415bd40 0x7f94d40d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51054/0 (socket says 172.21.15.6:51054) 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94d9008640 1 -- 172.21.15.6:0/3670013736 learned_addr learned my addr 172.21.15.6:0/3670013736 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.807+0000 7f94d9809640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f94d40cfcd0 0x7f94d40ce540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45868/0 (socket says 172.21.15.6:45868) 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.808+0000 7f94d8807640 1 -- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d415bd40 msgr2=0x7f94d40d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.808+0000 7f94d8807640 1 --2- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d415bd40 0x7f94d40d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.808+0000 7f94d8807640 1 -- 172.21.15.6:0/3670013736 >> v1:172.21.15.92:6789/0 conn(0x7f94d40cfcd0 legacy=0x7f94d40ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.808+0000 7f94d8807640 1 -- 172.21.15.6:0/3670013736 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d40ccb10 con 0x7f94d40d1d90 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.808+0000 7f94d9008640 1 --2- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d415bd40 0x7f94d40d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.808+0000 7f94d8807640 1 --2- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d40d1d90 0x7f94d40cf790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f94c80097a0 tx=0x7f94c802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=745a54045b1c5665 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94d3fff640 1 -- 172.21.15.6:0/3670013736 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8035020 con 0x7f94d40d1d90 2024-03-18T14:42:02.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94d3fff640 1 -- 172.21.15.6:0/3670013736 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94c8031830 con 0x7f94d40d1d90 2024-03-18T14:42:02.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94d3fff640 1 -- 172.21.15.6:0/3670013736 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8031c00 con 0x7f94d40d1d90 2024-03-18T14:42:02.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 -- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d40d1d90 msgr2=0x7f94d40cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 --2- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d40d1d90 0x7f94d40cf790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f94c80097a0 tx=0x7f94c802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 -- 172.21.15.6:0/3670013736 shutdown_connections 2024-03-18T14:42:02.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 --2- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d40d1d90 0x7f94d40cf790 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 --2- 172.21.15.6:0/3670013736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d415bd40 0x7f94d40d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 -- 172.21.15.6:0/3670013736 >> 172.21.15.6:0/3670013736 conn(0x7f94d409ae20 msgr2=0x7f94d408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 -- 172.21.15.6:0/3670013736 shutdown_connections 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.809+0000 7f94db26c640 1 -- 172.21.15.6:0/3670013736 wait complete. 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.810+0000 7f94db26c640 1 Processor -- start 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.810+0000 7f94db26c640 1 -- start start 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94db26c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d40cfcd0 0x7f94d4152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94db26c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94d40d1d90 0x7f94d415e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94db26c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d415bd40 0x7f94d40c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94db26c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f94d40d0c40 con 0x7f94d40d1d90 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94db26c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f94d40d0ac0 con 0x7f94d415bd40 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94db26c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f94d40d0160 con 0x7f94d40cfcd0 2024-03-18T14:42:02.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94d8807640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94d40d1d90 0x7f94d415e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94d9008640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d40cfcd0 0x7f94d4152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94d9809640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d415bd40 0x7f94d40c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94d9008640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d40cfcd0 0x7f94d4152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51070/0 (socket says 172.21.15.6:51070) 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94d9008640 1 -- 172.21.15.6:0/3066727618 learned_addr learned my addr 172.21.15.6:0/3066727618 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.811+0000 7f94d8807640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94d40d1d90 0x7f94d415e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56756/0 (socket says 172.21.15.6:56756) 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d9809640 1 -- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d40cfcd0 msgr2=0x7f94d4152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d9809640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d40cfcd0 0x7f94d4152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d9809640 1 -- 172.21.15.6:0/3066727618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94d40d1d90 msgr2=0x7f94d415e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d9809640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94d40d1d90 0x7f94d415e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d9809640 1 -- 172.21.15.6:0/3066727618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d41534d0 con 0x7f94d415bd40 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d9008640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d40cfcd0 0x7f94d4152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d9809640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d415bd40 0x7f94d40c66d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c3c0 tx=0x7f94cc001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d1ffb640 1 -- 172.21.15.6:0/3066727618 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0074e0 con 0x7f94d415bd40 2024-03-18T14:42:02.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d1ffb640 1 -- 172.21.15.6:0/3066727618 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94cc007680 con 0x7f94d415bd40 2024-03-18T14:42:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94d1ffb640 1 -- 172.21.15.6:0/3066727618 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0049e0 con 0x7f94d415bd40 2024-03-18T14:42:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d40aed70 con 0x7f94d415bd40 2024-03-18T14:42:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.812+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d40af330 con 0x7f94d415bd40 2024-03-18T14:42:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.813+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d409fa90 con 0x7f94d415bd40 2024-03-18T14:42:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.813+0000 7f94d1ffb640 1 -- 172.21.15.6:0/3066727618 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f94cc01a020 con 0x7f94d415bd40 2024-03-18T14:42:02.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.814+0000 7f94d1ffb640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f94c006bc00 0x7f94c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:02.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.814+0000 7f94d1ffb640 1 -- 172.21.15.6:0/3066727618 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f94cc08a410 con 0x7f94d415bd40 2024-03-18T14:42:02.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.814+0000 7f94d9008640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f94c006bc00 0x7f94c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:02.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.814+0000 7f94d9008640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f94c006bc00 0x7f94c006e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009fd0 tx=0x7f94c400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:02.823+0000 7f94d1ffb640 1 -- 172.21.15.6:0/3066727618 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f94cc0584e0 con 0x7f94d415bd40 2024-03-18T14:42:03.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.000+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f94d404f620 con 0x7f94c006bc00 2024-03-18T14:42:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.008+0000 7f94d1ffb640 1 -- 172.21.15.6:0/3066727618 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f94d404f620 con 0x7f94c006bc00 2024-03-18T14:42:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:03.015 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f94c006bc00 msgr2=0x7f94c006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f94c006bc00 0x7f94c006e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009fd0 tx=0x7f94c400b040 comp rx=0 tx=0).stop 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d415bd40 msgr2=0x7f94d40c66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d415bd40 0x7f94d40c66d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c3c0 tx=0x7f94cc001e70 comp rx=0 tx=0).stop 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 shutdown_connections 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f94c006bc00 0x7f94c006e0c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d415bd40 0x7f94d40c66d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94d40d1d90 0x7f94d415e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 --2- 172.21.15.6:0/3066727618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94d40cfcd0 0x7f94d4152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.011+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 >> 172.21.15.6:0/3066727618 conn(0x7f94d409ae20 msgr2=0x7f94d4071080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.012+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 shutdown_connections 2024-03-18T14:42:03.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.012+0000 7f94db26c640 1 -- 172.21.15.6:0/3066727618 wait complete. 2024-03-18T14:42:03.024 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-18T14:42:01.385243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002586"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779658+0000","last_change":"2024-03-18T14:41:34.533975+0000","last_active":"2024-03-18T14:41:40.779658+0000","last_peered":"2024-03-18T14:41:40.779658+0000","last_clean":"2024-03-18T14:41:40.779658+0000","last_became_active":"2024-03-18T14:41:34.533697+0000","last_became_peered":"2024-03-18T14:41:34.533697+0000","last_unstale":"2024-03-18T14:41:40.779658+0000","last_undegraded":"2024-03-18T14:41:40.779658+0000","last_fullsized":"2024-03-18T14:41:40.779658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:35.555471+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:35.555193+0000","last_became_peered":"2024-03-18T14:41:35.555193+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563408+0000","last_change":"2024-03-18T14:41:39.604430+0000","last_active":"2024-03-18T14:41:40.563408+0000","last_peered":"2024-03-18T14:41:40.563408+0000","last_clean":"2024-03-18T14:41:40.563408+0000","last_became_active":"2024-03-18T14:41:39.603958+0000","last_became_peered":"2024-03-18T14:41:39.603958+0000","last_unstale":"2024-03-18T14:41:40.563408+0000","last_undegraded":"2024-03-18T14:41:40.563408+0000","last_fullsized":"2024-03-18T14:41:40.563408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:17.770515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564369+0000","last_change":"2024-03-18T14:41:34.535172+0000","last_active":"2024-03-18T14:41:40.564369+0000","last_peered":"2024-03-18T14:41:40.564369+0000","last_clean":"2024-03-18T14:41:40.564369+0000","last_became_active":"2024-03-18T14:41:34.534849+0000","last_became_peered":"2024-03-18T14:41:34.534849+0000","last_unstale":"2024-03-18T14:41:40.564369+0000","last_undegraded":"2024-03-18T14:41:40.564369+0000","last_fullsized":"2024-03-18T14:41:40.564369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779347+0000","last_change":"2024-03-18T14:41:35.553407+0000","last_active":"2024-03-18T14:41:40.779347+0000","last_peered":"2024-03-18T14:41:40.779347+0000","last_clean":"2024-03-18T14:41:40.779347+0000","last_became_active":"2024-03-18T14:41:35.553091+0000","last_became_peered":"2024-03-18T14:41:35.553091+0000","last_unstale":"2024-03-18T14:41:40.779347+0000","last_undegraded":"2024-03-18T14:41:40.779347+0000","last_fullsized":"2024-03-18T14:41:40.779347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559102+0000","last_change":"2024-03-18T14:41:39.579025+0000","last_active":"2024-03-18T14:41:40.559102+0000","last_peered":"2024-03-18T14:41:40.559102+0000","last_clean":"2024-03-18T14:41:40.559102+0000","last_became_active":"2024-03-18T14:41:39.578650+0000","last_became_peered":"2024-03-18T14:41:39.578650+0000","last_unstale":"2024-03-18T14:41:40.559102+0000","last_undegraded":"2024-03-18T14:41:40.559102+0000","last_fullsized":"2024-03-18T14:41:40.559102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:53.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778929+0000","last_change":"2024-03-18T14:41:34.536627+0000","last_active":"2024-03-18T14:41:40.778929+0000","last_peered":"2024-03-18T14:41:40.778929+0000","last_clean":"2024-03-18T14:41:40.778929+0000","last_became_active":"2024-03-18T14:41:34.536136+0000","last_became_peered":"2024-03-18T14:41:34.536136+0000","last_unstale":"2024-03-18T14:41:40.778929+0000","last_undegraded":"2024-03-18T14:41:40.778929+0000","last_fullsized":"2024-03-18T14:41:40.778929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:12.036369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559536+0000","last_change":"2024-03-18T14:41:35.544918+0000","last_active":"2024-03-18T14:41:40.559536+0000","last_peered":"2024-03-18T14:41:40.559536+0000","last_clean":"2024-03-18T14:41:40.559536+0000","last_became_active":"2024-03-18T14:41:35.544612+0000","last_became_peered":"2024-03-18T14:41:35.544612+0000","last_unstale":"2024-03-18T14:41:40.559536+0000","last_undegraded":"2024-03-18T14:41:40.559536+0000","last_fullsized":"2024-03-18T14:41:40.559536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560251+0000","last_change":"2024-03-18T14:41:39.580665+0000","last_active":"2024-03-18T14:41:40.560251+0000","last_peered":"2024-03-18T14:41:40.560251+0000","last_clean":"2024-03-18T14:41:40.560251+0000","last_became_active":"2024-03-18T14:41:39.580231+0000","last_became_peered":"2024-03-18T14:41:39.580231+0000","last_unstale":"2024-03-18T14:41:40.560251+0000","last_undegraded":"2024-03-18T14:41:40.560251+0000","last_fullsized":"2024-03-18T14:41:40.560251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778491+0000","last_change":"2024-03-18T14:41:34.531008+0000","last_active":"2024-03-18T14:41:40.778491+0000","last_peered":"2024-03-18T14:41:40.778491+0000","last_clean":"2024-03-18T14:41:40.778491+0000","last_became_active":"2024-03-18T14:41:34.530587+0000","last_became_peered":"2024-03-18T14:41:34.530587+0000","last_unstale":"2024-03-18T14:41:40.778491+0000","last_undegraded":"2024-03-18T14:41:40.778491+0000","last_fullsized":"2024-03-18T14:41:40.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560259+0000","last_change":"2024-03-18T14:41:35.542756+0000","last_active":"2024-03-18T14:41:40.560259+0000","last_peered":"2024-03-18T14:41:40.560259+0000","last_clean":"2024-03-18T14:41:40.560259+0000","last_became_active":"2024-03-18T14:41:35.542486+0000","last_became_peered":"2024-03-18T14:41:35.542486+0000","last_unstale":"2024-03-18T14:41:40.560259+0000","last_undegraded":"2024-03-18T14:41:40.560259+0000","last_fullsized":"2024-03-18T14:41:40.560259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559437+0000","last_change":"2024-03-18T14:41:39.578229+0000","last_active":"2024-03-18T14:41:40.559437+0000","last_peered":"2024-03-18T14:41:40.559437+0000","last_clean":"2024-03-18T14:41:40.559437+0000","last_became_active":"2024-03-18T14:41:39.577812+0000","last_became_peered":"2024-03-18T14:41:39.577812+0000","last_unstale":"2024-03-18T14:41:40.559437+0000","last_undegraded":"2024-03-18T14:41:40.559437+0000","last_fullsized":"2024-03-18T14:41:40.559437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:50.123586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778041+0000","last_change":"2024-03-18T14:41:35.549686+0000","last_active":"2024-03-18T14:41:40.778041+0000","last_peered":"2024-03-18T14:41:40.778041+0000","last_clean":"2024-03-18T14:41:40.778041+0000","last_became_active":"2024-03-18T14:41:35.549240+0000","last_became_peered":"2024-03-18T14:41:35.549240+0000","last_unstale":"2024-03-18T14:41:40.778041+0000","last_undegraded":"2024-03-18T14:41:40.778041+0000","last_fullsized":"2024-03-18T14:41:40.778041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778484+0000","last_change":"2024-03-18T14:41:34.530798+0000","last_active":"2024-03-18T14:41:40.778484+0000","last_peered":"2024-03-18T14:41:40.778484+0000","last_clean":"2024-03-18T14:41:40.778484+0000","last_became_active":"2024-03-18T14:41:34.530415+0000","last_became_peered":"2024-03-18T14:41:34.530415+0000","last_unstale":"2024-03-18T14:41:40.778484+0000","last_undegraded":"2024-03-18T14:41:40.778484+0000","last_fullsized":"2024-03-18T14:41:40.778484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560204+0000","last_change":"2024-03-18T14:41:39.584982+0000","last_active":"2024-03-18T14:41:40.560204+0000","last_peered":"2024-03-18T14:41:40.560204+0000","last_clean":"2024-03-18T14:41:40.560204+0000","last_became_active":"2024-03-18T14:41:39.584630+0000","last_became_peered":"2024-03-18T14:41:39.584630+0000","last_unstale":"2024-03-18T14:41:40.560204+0000","last_undegraded":"2024-03-18T14:41:40.560204+0000","last_fullsized":"2024-03-18T14:41:40.560204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563830+0000","last_change":"2024-03-18T14:41:35.552305+0000","last_active":"2024-03-18T14:41:40.563830+0000","last_peered":"2024-03-18T14:41:40.563830+0000","last_clean":"2024-03-18T14:41:40.563830+0000","last_became_active":"2024-03-18T14:41:35.552031+0000","last_became_peered":"2024-03-18T14:41:35.552031+0000","last_unstale":"2024-03-18T14:41:40.563830+0000","last_undegraded":"2024-03-18T14:41:40.563830+0000","last_fullsized":"2024-03-18T14:41:40.563830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779939+0000","last_change":"2024-03-18T14:41:34.530930+0000","last_active":"2024-03-18T14:41:40.779939+0000","last_peered":"2024-03-18T14:41:40.779939+0000","last_clean":"2024-03-18T14:41:40.779939+0000","last_became_active":"2024-03-18T14:41:34.530645+0000","last_became_peered":"2024-03-18T14:41:34.530645+0000","last_unstale":"2024-03-18T14:41:40.779939+0000","last_undegraded":"2024-03-18T14:41:40.779939+0000","last_fullsized":"2024-03-18T14:41:40.779939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559457+0000","last_change":"2024-03-18T14:41:39.585926+0000","last_active":"2024-03-18T14:41:40.559457+0000","last_peered":"2024-03-18T14:41:40.559457+0000","last_clean":"2024-03-18T14:41:40.559457+0000","last_became_active":"2024-03-18T14:41:39.585465+0000","last_became_peered":"2024-03-18T14:41:39.585465+0000","last_unstale":"2024-03-18T14:41:40.559457+0000","last_undegraded":"2024-03-18T14:41:40.559457+0000","last_fullsized":"2024-03-18T14:41:40.559457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563386+0000","last_change":"2024-03-18T14:41:35.541836+0000","last_active":"2024-03-18T14:41:40.563386+0000","last_peered":"2024-03-18T14:41:40.563386+0000","last_clean":"2024-03-18T14:41:40.563386+0000","last_became_active":"2024-03-18T14:41:35.541532+0000","last_became_peered":"2024-03-18T14:41:35.541532+0000","last_unstale":"2024-03-18T14:41:40.563386+0000","last_undegraded":"2024-03-18T14:41:40.563386+0000","last_fullsized":"2024-03-18T14:41:40.563386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778143+0000","last_change":"2024-03-18T14:41:34.529169+0000","last_active":"2024-03-18T14:41:40.778143+0000","last_peered":"2024-03-18T14:41:40.778143+0000","last_clean":"2024-03-18T14:41:40.778143+0000","last_became_active":"2024-03-18T14:41:34.526870+0000","last_became_peered":"2024-03-18T14:41:34.526870+0000","last_unstale":"2024-03-18T14:41:40.778143+0000","last_undegraded":"2024-03-18T14:41:40.778143+0000","last_fullsized":"2024-03-18T14:41:40.778143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779714+0000","last_change":"2024-03-18T14:41:39.603928+0000","last_active":"2024-03-18T14:41:40.779714+0000","last_peered":"2024-03-18T14:41:40.779714+0000","last_clean":"2024-03-18T14:41:40.779714+0000","last_became_active":"2024-03-18T14:41:39.603236+0000","last_became_peered":"2024-03-18T14:41:39.603236+0000","last_unstale":"2024-03-18T14:41:40.779714+0000","last_undegraded":"2024-03-18T14:41:40.779714+0000","last_fullsized":"2024-03-18T14:41:40.779714+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559885+0000","last_change":"2024-03-18T14:41:35.538978+0000","last_active":"2024-03-18T14:41:40.559885+0000","last_peered":"2024-03-18T14:41:40.559885+0000","last_clean":"2024-03-18T14:41:40.559885+0000","last_became_active":"2024-03-18T14:41:35.538570+0000","last_became_peered":"2024-03-18T14:41:35.538570+0000","last_unstale":"2024-03-18T14:41:40.559885+0000","last_undegraded":"2024-03-18T14:41:40.559885+0000","last_fullsized":"2024-03-18T14:41:40.559885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:38.574781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777921+0000","last_change":"2024-03-18T14:41:34.537652+0000","last_active":"2024-03-18T14:41:40.777921+0000","last_peered":"2024-03-18T14:41:40.777921+0000","last_clean":"2024-03-18T14:41:40.777921+0000","last_became_active":"2024-03-18T14:41:34.537411+0000","last_became_peered":"2024-03-18T14:41:34.537411+0000","last_unstale":"2024-03-18T14:41:40.777921+0000","last_undegraded":"2024-03-18T14:41:40.777921+0000","last_fullsized":"2024-03-18T14:41:40.777921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560190+0000","last_change":"2024-03-18T14:41:39.580821+0000","last_active":"2024-03-18T14:41:40.560190+0000","last_peered":"2024-03-18T14:41:40.560190+0000","last_clean":"2024-03-18T14:41:40.560190+0000","last_became_active":"2024-03-18T14:41:39.580317+0000","last_became_peered":"2024-03-18T14:41:39.580317+0000","last_unstale":"2024-03-18T14:41:40.560190+0000","last_undegraded":"2024-03-18T14:41:40.560190+0000","last_fullsized":"2024-03-18T14:41:40.560190+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778799+0000","last_change":"2024-03-18T14:41:35.552139+0000","last_active":"2024-03-18T14:41:40.778799+0000","last_peered":"2024-03-18T14:41:40.778799+0000","last_clean":"2024-03-18T14:41:40.778799+0000","last_became_active":"2024-03-18T14:41:35.551809+0000","last_became_peered":"2024-03-18T14:41:35.551809+0000","last_unstale":"2024-03-18T14:41:40.778799+0000","last_undegraded":"2024-03-18T14:41:40.778799+0000","last_fullsized":"2024-03-18T14:41:40.778799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779124+0000","last_change":"2024-03-18T14:41:34.534282+0000","last_active":"2024-03-18T14:41:40.779124+0000","last_peered":"2024-03-18T14:41:40.779124+0000","last_clean":"2024-03-18T14:41:40.779124+0000","last_became_active":"2024-03-18T14:41:34.533937+0000","last_became_peered":"2024-03-18T14:41:34.533937+0000","last_unstale":"2024-03-18T14:41:40.779124+0000","last_undegraded":"2024-03-18T14:41:40.779124+0000","last_fullsized":"2024-03-18T14:41:40.779124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559601+0000","last_change":"2024-03-18T14:41:39.602565+0000","last_active":"2024-03-18T14:41:40.559601+0000","last_peered":"2024-03-18T14:41:40.559601+0000","last_clean":"2024-03-18T14:41:40.559601+0000","last_became_active":"2024-03-18T14:41:39.602164+0000","last_became_peered":"2024-03-18T14:41:39.602164+0000","last_unstale":"2024-03-18T14:41:40.559601+0000","last_undegraded":"2024-03-18T14:41:40.559601+0000","last_fullsized":"2024-03-18T14:41:40.559601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564632+0000","last_change":"2024-03-18T14:41:35.547600+0000","last_active":"2024-03-18T14:41:40.564632+0000","last_peered":"2024-03-18T14:41:40.564632+0000","last_clean":"2024-03-18T14:41:40.564632+0000","last_became_active":"2024-03-18T14:41:35.547324+0000","last_became_peered":"2024-03-18T14:41:35.547324+0000","last_unstale":"2024-03-18T14:41:40.564632+0000","last_undegraded":"2024-03-18T14:41:40.564632+0000","last_fullsized":"2024-03-18T14:41:40.564632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779055+0000","last_change":"2024-03-18T14:41:34.537441+0000","last_active":"2024-03-18T14:41:40.779055+0000","last_peered":"2024-03-18T14:41:40.779055+0000","last_clean":"2024-03-18T14:41:40.779055+0000","last_became_active":"2024-03-18T14:41:34.536599+0000","last_became_peered":"2024-03-18T14:41:34.536599+0000","last_unstale":"2024-03-18T14:41:40.779055+0000","last_undegraded":"2024-03-18T14:41:40.779055+0000","last_fullsized":"2024-03-18T14:41:40.779055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563656+0000","last_change":"2024-03-18T14:41:39.604429+0000","last_active":"2024-03-18T14:41:40.563656+0000","last_peered":"2024-03-18T14:41:40.563656+0000","last_clean":"2024-03-18T14:41:40.563656+0000","last_became_active":"2024-03-18T14:41:39.604046+0000","last_became_peered":"2024-03-18T14:41:39.604046+0000","last_unstale":"2024-03-18T14:41:40.563656+0000","last_undegraded":"2024-03-18T14:41:40.563656+0000","last_fullsized":"2024-03-18T14:41:40.563656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:21.835327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778649+0000","last_change":"2024-03-18T14:41:35.539912+0000","last_active":"2024-03-18T14:41:40.778649+0000","last_peered":"2024-03-18T14:41:40.778649+0000","last_clean":"2024-03-18T14:41:40.778649+0000","last_became_active":"2024-03-18T14:41:35.538765+0000","last_became_peered":"2024-03-18T14:41:35.538765+0000","last_unstale":"2024-03-18T14:41:40.778649+0000","last_undegraded":"2024-03-18T14:41:40.778649+0000","last_fullsized":"2024-03-18T14:41:40.778649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779021+0000","last_change":"2024-03-18T14:41:34.537358+0000","last_active":"2024-03-18T14:41:40.779021+0000","last_peered":"2024-03-18T14:41:40.779021+0000","last_clean":"2024-03-18T14:41:40.779021+0000","last_became_active":"2024-03-18T14:41:34.537020+0000","last_became_peered":"2024-03-18T14:41:34.537020+0000","last_unstale":"2024-03-18T14:41:40.779021+0000","last_undegraded":"2024-03-18T14:41:40.779021+0000","last_fullsized":"2024-03-18T14:41:40.779021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:24.398074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778734+0000","last_change":"2024-03-18T14:41:39.601563+0000","last_active":"2024-03-18T14:41:40.778734+0000","last_peered":"2024-03-18T14:41:40.778734+0000","last_clean":"2024-03-18T14:41:40.778734+0000","last_became_active":"2024-03-18T14:41:39.600878+0000","last_became_peered":"2024-03-18T14:41:39.600878+0000","last_unstale":"2024-03-18T14:41:40.778734+0000","last_undegraded":"2024-03-18T14:41:40.778734+0000","last_fullsized":"2024-03-18T14:41:40.778734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:41.241691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560348+0000","last_change":"2024-03-18T14:41:35.551415+0000","last_active":"2024-03-18T14:41:40.560348+0000","last_peered":"2024-03-18T14:41:40.560348+0000","last_clean":"2024-03-18T14:41:40.560348+0000","last_became_active":"2024-03-18T14:41:35.550751+0000","last_became_peered":"2024-03-18T14:41:35.550751+0000","last_unstale":"2024-03-18T14:41:40.560348+0000","last_undegraded":"2024-03-18T14:41:40.560348+0000","last_fullsized":"2024-03-18T14:41:40.560348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778749+0000","last_change":"2024-03-18T14:41:34.534249+0000","last_active":"2024-03-18T14:41:40.778749+0000","last_peered":"2024-03-18T14:41:40.778749+0000","last_clean":"2024-03-18T14:41:40.778749+0000","last_became_active":"2024-03-18T14:41:34.533913+0000","last_became_peered":"2024-03-18T14:41:34.533913+0000","last_unstale":"2024-03-18T14:41:40.778749+0000","last_undegraded":"2024-03-18T14:41:40.778749+0000","last_fullsized":"2024-03-18T14:41:40.778749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:39.582572+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:39.581652+0000","last_became_peered":"2024-03-18T14:41:39.581652+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560327+0000","last_change":"2024-03-18T14:41:35.561297+0000","last_active":"2024-03-18T14:41:40.560327+0000","last_peered":"2024-03-18T14:41:40.560327+0000","last_clean":"2024-03-18T14:41:40.560327+0000","last_became_active":"2024-03-18T14:41:35.558687+0000","last_became_peered":"2024-03-18T14:41:35.558687+0000","last_unstale":"2024-03-18T14:41:40.560327+0000","last_undegraded":"2024-03-18T14:41:40.560327+0000","last_fullsized":"2024-03-18T14:41:40.560327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:25.399581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559780+0000","last_change":"2024-03-18T14:41:34.525754+0000","last_active":"2024-03-18T14:41:40.559780+0000","last_peered":"2024-03-18T14:41:40.559780+0000","last_clean":"2024-03-18T14:41:40.559780+0000","last_became_active":"2024-03-18T14:41:34.525453+0000","last_became_peered":"2024-03-18T14:41:34.525453+0000","last_unstale":"2024-03-18T14:41:40.559780+0000","last_undegraded":"2024-03-18T14:41:40.559780+0000","last_fullsized":"2024-03-18T14:41:40.559780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778594+0000","last_change":"2024-03-18T14:41:39.584128+0000","last_active":"2024-03-18T14:41:40.778594+0000","last_peered":"2024-03-18T14:41:40.778594+0000","last_clean":"2024-03-18T14:41:40.778594+0000","last_became_active":"2024-03-18T14:41:39.582472+0000","last_became_peered":"2024-03-18T14:41:39.582472+0000","last_unstale":"2024-03-18T14:41:40.778594+0000","last_undegraded":"2024-03-18T14:41:40.778594+0000","last_fullsized":"2024-03-18T14:41:40.778594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778049+0000","last_change":"2024-03-18T14:41:35.540356+0000","last_active":"2024-03-18T14:41:40.778049+0000","last_peered":"2024-03-18T14:41:40.778049+0000","last_clean":"2024-03-18T14:41:40.778049+0000","last_became_active":"2024-03-18T14:41:35.540060+0000","last_became_peered":"2024-03-18T14:41:35.540060+0000","last_unstale":"2024-03-18T14:41:40.778049+0000","last_undegraded":"2024-03-18T14:41:40.778049+0000","last_fullsized":"2024-03-18T14:41:40.778049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778106+0000","last_change":"2024-03-18T14:41:34.529076+0000","last_active":"2024-03-18T14:41:40.778106+0000","last_peered":"2024-03-18T14:41:40.778106+0000","last_clean":"2024-03-18T14:41:40.778106+0000","last_became_active":"2024-03-18T14:41:34.528794+0000","last_became_peered":"2024-03-18T14:41:34.528794+0000","last_unstale":"2024-03-18T14:41:40.778106+0000","last_undegraded":"2024-03-18T14:41:40.778106+0000","last_fullsized":"2024-03-18T14:41:40.778106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779855+0000","last_change":"2024-03-18T14:41:39.581385+0000","last_active":"2024-03-18T14:41:40.779855+0000","last_peered":"2024-03-18T14:41:40.779855+0000","last_clean":"2024-03-18T14:41:40.779855+0000","last_became_active":"2024-03-18T14:41:39.580972+0000","last_became_peered":"2024-03-18T14:41:39.580972+0000","last_unstale":"2024-03-18T14:41:40.779855+0000","last_undegraded":"2024-03-18T14:41:40.779855+0000","last_fullsized":"2024-03-18T14:41:40.779855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778334+0000","last_change":"2024-03-18T14:41:35.552692+0000","last_active":"2024-03-18T14:41:40.778334+0000","last_peered":"2024-03-18T14:41:40.778334+0000","last_clean":"2024-03-18T14:41:40.778334+0000","last_became_active":"2024-03-18T14:41:35.552340+0000","last_became_peered":"2024-03-18T14:41:35.552340+0000","last_unstale":"2024-03-18T14:41:40.778334+0000","last_undegraded":"2024-03-18T14:41:40.778334+0000","last_fullsized":"2024-03-18T14:41:40.778334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778015+0000","last_change":"2024-03-18T14:41:34.531973+0000","last_active":"2024-03-18T14:41:40.778015+0000","last_peered":"2024-03-18T14:41:40.778015+0000","last_clean":"2024-03-18T14:41:40.778015+0000","last_became_active":"2024-03-18T14:41:34.531796+0000","last_became_peered":"2024-03-18T14:41:34.531796+0000","last_unstale":"2024-03-18T14:41:40.778015+0000","last_undegraded":"2024-03-18T14:41:40.778015+0000","last_fullsized":"2024-03-18T14:41:40.778015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778621+0000","last_change":"2024-03-18T14:41:39.601287+0000","last_active":"2024-03-18T14:41:40.778621+0000","last_peered":"2024-03-18T14:41:40.778621+0000","last_clean":"2024-03-18T14:41:40.778621+0000","last_became_active":"2024-03-18T14:41:39.600855+0000","last_became_peered":"2024-03-18T14:41:39.600855+0000","last_unstale":"2024-03-18T14:41:40.778621+0000","last_undegraded":"2024-03-18T14:41:40.778621+0000","last_fullsized":"2024-03-18T14:41:40.778621+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563301+0000","last_change":"2024-03-18T14:41:35.547848+0000","last_active":"2024-03-18T14:41:40.563301+0000","last_peered":"2024-03-18T14:41:40.563301+0000","last_clean":"2024-03-18T14:41:40.563301+0000","last_became_active":"2024-03-18T14:41:35.547437+0000","last_became_peered":"2024-03-18T14:41:35.547437+0000","last_unstale":"2024-03-18T14:41:40.563301+0000","last_undegraded":"2024-03-18T14:41:40.563301+0000","last_fullsized":"2024-03-18T14:41:40.563301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559408+0000","last_change":"2024-03-18T14:41:34.528039+0000","last_active":"2024-03-18T14:41:40.559408+0000","last_peered":"2024-03-18T14:41:40.559408+0000","last_clean":"2024-03-18T14:41:40.559408+0000","last_became_active":"2024-03-18T14:41:34.527756+0000","last_became_peered":"2024-03-18T14:41:34.527756+0000","last_unstale":"2024-03-18T14:41:40.559408+0000","last_undegraded":"2024-03-18T14:41:40.559408+0000","last_fullsized":"2024-03-18T14:41:40.559408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778101+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:41:40.778101+0000","last_peered":"2024-03-18T14:41:40.778101+0000","last_clean":"2024-03-18T14:41:40.778101+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:41:40.778101+0000","last_undegraded":"2024-03-18T14:41:40.778101+0000","last_fullsized":"2024-03-18T14:41:40.778101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778699+0000","last_change":"2024-03-18T14:41:39.602108+0000","last_active":"2024-03-18T14:41:40.778699+0000","last_peered":"2024-03-18T14:41:40.778699+0000","last_clean":"2024-03-18T14:41:40.778699+0000","last_became_active":"2024-03-18T14:41:39.601732+0000","last_became_peered":"2024-03-18T14:41:39.601732+0000","last_unstale":"2024-03-18T14:41:40.778699+0000","last_undegraded":"2024-03-18T14:41:40.778699+0000","last_fullsized":"2024-03-18T14:41:40.778699+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778841+0000","last_change":"2024-03-18T14:41:35.544776+0000","last_active":"2024-03-18T14:41:40.778841+0000","last_peered":"2024-03-18T14:41:40.778841+0000","last_clean":"2024-03-18T14:41:40.778841+0000","last_became_active":"2024-03-18T14:41:35.544498+0000","last_became_peered":"2024-03-18T14:41:35.544498+0000","last_unstale":"2024-03-18T14:41:40.778841+0000","last_undegraded":"2024-03-18T14:41:40.778841+0000","last_fullsized":"2024-03-18T14:41:40.778841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780098+0000","last_change":"2024-03-18T14:41:34.531645+0000","last_active":"2024-03-18T14:41:40.780098+0000","last_peered":"2024-03-18T14:41:40.780098+0000","last_clean":"2024-03-18T14:41:40.780098+0000","last_became_active":"2024-03-18T14:41:34.530361+0000","last_became_peered":"2024-03-18T14:41:34.530361+0000","last_unstale":"2024-03-18T14:41:40.780098+0000","last_undegraded":"2024-03-18T14:41:40.780098+0000","last_fullsized":"2024-03-18T14:41:40.780098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778063+0000","last_change":"2024-03-18T14:41:39.601190+0000","last_active":"2024-03-18T14:41:40.778063+0000","last_peered":"2024-03-18T14:41:40.778063+0000","last_clean":"2024-03-18T14:41:40.778063+0000","last_became_active":"2024-03-18T14:41:39.600809+0000","last_became_peered":"2024-03-18T14:41:39.600809+0000","last_unstale":"2024-03-18T14:41:40.778063+0000","last_undegraded":"2024-03-18T14:41:40.778063+0000","last_fullsized":"2024-03-18T14:41:40.778063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559630+0000","last_change":"2024-03-18T14:41:35.549443+0000","last_active":"2024-03-18T14:41:40.559630+0000","last_peered":"2024-03-18T14:41:40.559630+0000","last_clean":"2024-03-18T14:41:40.559630+0000","last_became_active":"2024-03-18T14:41:35.546376+0000","last_became_peered":"2024-03-18T14:41:35.546376+0000","last_unstale":"2024-03-18T14:41:40.559630+0000","last_undegraded":"2024-03-18T14:41:40.559630+0000","last_fullsized":"2024-03-18T14:41:40.559630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779349+0000","last_change":"2024-03-18T14:41:34.528408+0000","last_active":"2024-03-18T14:41:40.779349+0000","last_peered":"2024-03-18T14:41:40.779349+0000","last_clean":"2024-03-18T14:41:40.779349+0000","last_became_active":"2024-03-18T14:41:34.527981+0000","last_became_peered":"2024-03-18T14:41:34.527981+0000","last_unstale":"2024-03-18T14:41:40.779349+0000","last_undegraded":"2024-03-18T14:41:40.779349+0000","last_fullsized":"2024-03-18T14:41:40.779349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778976+0000","last_change":"2024-03-18T14:41:39.586214+0000","last_active":"2024-03-18T14:41:40.778976+0000","last_peered":"2024-03-18T14:41:40.778976+0000","last_clean":"2024-03-18T14:41:40.778976+0000","last_became_active":"2024-03-18T14:41:39.585773+0000","last_became_peered":"2024-03-18T14:41:39.585773+0000","last_unstale":"2024-03-18T14:41:40.778976+0000","last_undegraded":"2024-03-18T14:41:40.778976+0000","last_fullsized":"2024-03-18T14:41:40.778976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:52:45.131607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778850+0000","last_change":"2024-03-18T14:41:35.553371+0000","last_active":"2024-03-18T14:41:40.778850+0000","last_peered":"2024-03-18T14:41:40.778850+0000","last_clean":"2024-03-18T14:41:40.778850+0000","last_became_active":"2024-03-18T14:41:35.552906+0000","last_became_peered":"2024-03-18T14:41:35.552906+0000","last_unstale":"2024-03-18T14:41:40.778850+0000","last_undegraded":"2024-03-18T14:41:40.778850+0000","last_fullsized":"2024-03-18T14:41:40.778850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778853+0000","last_change":"2024-03-18T14:41:34.532843+0000","last_active":"2024-03-18T14:41:40.778853+0000","last_peered":"2024-03-18T14:41:40.778853+0000","last_clean":"2024-03-18T14:41:40.778853+0000","last_became_active":"2024-03-18T14:41:34.532183+0000","last_became_peered":"2024-03-18T14:41:34.532183+0000","last_unstale":"2024-03-18T14:41:40.778853+0000","last_undegraded":"2024-03-18T14:41:40.778853+0000","last_fullsized":"2024-03-18T14:41:40.778853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559016+0000","last_change":"2024-03-18T14:41:39.603278+0000","last_active":"2024-03-18T14:41:40.559016+0000","last_peered":"2024-03-18T14:41:40.559016+0000","last_clean":"2024-03-18T14:41:40.559016+0000","last_became_active":"2024-03-18T14:41:39.602927+0000","last_became_peered":"2024-03-18T14:41:39.602927+0000","last_unstale":"2024-03-18T14:41:40.559016+0000","last_undegraded":"2024-03-18T14:41:40.559016+0000","last_fullsized":"2024-03-18T14:41:40.559016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778747+0000","last_change":"2024-03-18T14:41:35.552924+0000","last_active":"2024-03-18T14:41:40.778747+0000","last_peered":"2024-03-18T14:41:40.778747+0000","last_clean":"2024-03-18T14:41:40.778747+0000","last_became_active":"2024-03-18T14:41:35.552514+0000","last_became_peered":"2024-03-18T14:41:35.552514+0000","last_unstale":"2024-03-18T14:41:40.778747+0000","last_undegraded":"2024-03-18T14:41:40.778747+0000","last_fullsized":"2024-03-18T14:41:40.778747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560464+0000","last_change":"2024-03-18T14:41:34.534565+0000","last_active":"2024-03-18T14:41:40.560464+0000","last_peered":"2024-03-18T14:41:40.560464+0000","last_clean":"2024-03-18T14:41:40.560464+0000","last_became_active":"2024-03-18T14:41:34.534302+0000","last_became_peered":"2024-03-18T14:41:34.534302+0000","last_unstale":"2024-03-18T14:41:40.560464+0000","last_undegraded":"2024-03-18T14:41:40.560464+0000","last_fullsized":"2024-03-18T14:41:40.560464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559373+0000","last_change":"2024-03-18T14:41:39.580698+0000","last_active":"2024-03-18T14:41:40.559373+0000","last_peered":"2024-03-18T14:41:40.559373+0000","last_clean":"2024-03-18T14:41:40.559373+0000","last_became_active":"2024-03-18T14:41:39.580161+0000","last_became_peered":"2024-03-18T14:41:39.580161+0000","last_unstale":"2024-03-18T14:41:40.559373+0000","last_undegraded":"2024-03-18T14:41:40.559373+0000","last_fullsized":"2024-03-18T14:41:40.559373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:03.265239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778352+0000","last_change":"2024-03-18T14:41:35.547876+0000","last_active":"2024-03-18T14:41:40.778352+0000","last_peered":"2024-03-18T14:41:40.778352+0000","last_clean":"2024-03-18T14:41:40.778352+0000","last_became_active":"2024-03-18T14:41:35.547615+0000","last_became_peered":"2024-03-18T14:41:35.547615+0000","last_unstale":"2024-03-18T14:41:40.778352+0000","last_undegraded":"2024-03-18T14:41:40.778352+0000","last_fullsized":"2024-03-18T14:41:40.778352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777856+0000","last_change":"2024-03-18T14:41:34.526247+0000","last_active":"2024-03-18T14:41:40.777856+0000","last_peered":"2024-03-18T14:41:40.777856+0000","last_clean":"2024-03-18T14:41:40.777856+0000","last_became_active":"2024-03-18T14:41:34.525952+0000","last_became_peered":"2024-03-18T14:41:34.525952+0000","last_unstale":"2024-03-18T14:41:40.777856+0000","last_undegraded":"2024-03-18T14:41:40.777856+0000","last_fullsized":"2024-03-18T14:41:40.777856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:39.580945+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:39.580421+0000","last_became_peered":"2024-03-18T14:41:39.580421+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778016+0000","last_change":"2024-03-18T14:41:35.554425+0000","last_active":"2024-03-18T14:41:40.778016+0000","last_peered":"2024-03-18T14:41:40.778016+0000","last_clean":"2024-03-18T14:41:40.778016+0000","last_became_active":"2024-03-18T14:41:35.554164+0000","last_became_peered":"2024-03-18T14:41:35.554164+0000","last_unstale":"2024-03-18T14:41:40.778016+0000","last_undegraded":"2024-03-18T14:41:40.778016+0000","last_fullsized":"2024-03-18T14:41:40.778016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778159+0000","last_change":"2024-03-18T14:41:34.528726+0000","last_active":"2024-03-18T14:41:40.778159+0000","last_peered":"2024-03-18T14:41:40.778159+0000","last_clean":"2024-03-18T14:41:40.778159+0000","last_became_active":"2024-03-18T14:41:34.528424+0000","last_became_peered":"2024-03-18T14:41:34.528424+0000","last_unstale":"2024-03-18T14:41:40.778159+0000","last_undegraded":"2024-03-18T14:41:40.778159+0000","last_fullsized":"2024-03-18T14:41:40.778159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564629+0000","last_change":"2024-03-18T14:41:39.602221+0000","last_active":"2024-03-18T14:41:40.564629+0000","last_peered":"2024-03-18T14:41:40.564629+0000","last_clean":"2024-03-18T14:41:40.564629+0000","last_became_active":"2024-03-18T14:41:39.601652+0000","last_became_peered":"2024-03-18T14:41:39.601652+0000","last_unstale":"2024-03-18T14:41:40.564629+0000","last_undegraded":"2024-03-18T14:41:40.564629+0000","last_fullsized":"2024-03-18T14:41:40.564629+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:00.363235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778400+0000","last_change":"2024-03-18T14:41:35.548474+0000","last_active":"2024-03-18T14:41:40.778400+0000","last_peered":"2024-03-18T14:41:40.778400+0000","last_clean":"2024-03-18T14:41:40.778400+0000","last_became_active":"2024-03-18T14:41:35.548193+0000","last_became_peered":"2024-03-18T14:41:35.548193+0000","last_unstale":"2024-03-18T14:41:40.778400+0000","last_undegraded":"2024-03-18T14:41:40.778400+0000","last_fullsized":"2024-03-18T14:41:40.778400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559701+0000","last_change":"2024-03-18T14:41:34.529770+0000","last_active":"2024-03-18T14:41:40.559701+0000","last_peered":"2024-03-18T14:41:40.559701+0000","last_clean":"2024-03-18T14:41:40.559701+0000","last_became_active":"2024-03-18T14:41:34.529499+0000","last_became_peered":"2024-03-18T14:41:34.529499+0000","last_unstale":"2024-03-18T14:41:40.559701+0000","last_undegraded":"2024-03-18T14:41:40.559701+0000","last_fullsized":"2024-03-18T14:41:40.559701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564084+0000","last_change":"2024-03-18T14:41:39.601622+0000","last_active":"2024-03-18T14:41:40.564084+0000","last_peered":"2024-03-18T14:41:40.564084+0000","last_clean":"2024-03-18T14:41:40.564084+0000","last_became_active":"2024-03-18T14:41:39.601209+0000","last_became_peered":"2024-03-18T14:41:39.601209+0000","last_unstale":"2024-03-18T14:41:40.564084+0000","last_undegraded":"2024-03-18T14:41:40.564084+0000","last_fullsized":"2024-03-18T14:41:40.564084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:11.443644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778881+0000","last_change":"2024-03-18T14:41:35.547642+0000","last_active":"2024-03-18T14:41:40.778881+0000","last_peered":"2024-03-18T14:41:40.778881+0000","last_clean":"2024-03-18T14:41:40.778881+0000","last_became_active":"2024-03-18T14:41:35.547354+0000","last_became_peered":"2024-03-18T14:41:35.547354+0000","last_unstale":"2024-03-18T14:41:40.778881+0000","last_undegraded":"2024-03-18T14:41:40.778881+0000","last_fullsized":"2024-03-18T14:41:40.778881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779200+0000","last_change":"2024-03-18T14:41:34.533998+0000","last_active":"2024-03-18T14:41:40.779200+0000","last_peered":"2024-03-18T14:41:40.779200+0000","last_clean":"2024-03-18T14:41:40.779200+0000","last_became_active":"2024-03-18T14:41:34.533597+0000","last_became_peered":"2024-03-18T14:41:34.533597+0000","last_unstale":"2024-03-18T14:41:40.779200+0000","last_undegraded":"2024-03-18T14:41:40.779200+0000","last_fullsized":"2024-03-18T14:41:40.779200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560211+0000","last_change":"2024-03-18T14:41:39.601126+0000","last_active":"2024-03-18T14:41:40.560211+0000","last_peered":"2024-03-18T14:41:40.560211+0000","last_clean":"2024-03-18T14:41:40.560211+0000","last_became_active":"2024-03-18T14:41:39.600751+0000","last_became_peered":"2024-03-18T14:41:39.600751+0000","last_unstale":"2024-03-18T14:41:40.560211+0000","last_undegraded":"2024-03-18T14:41:40.560211+0000","last_fullsized":"2024-03-18T14:41:40.560211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778816+0000","last_change":"2024-03-18T14:41:35.550195+0000","last_active":"2024-03-18T14:41:40.778816+0000","last_peered":"2024-03-18T14:41:40.778816+0000","last_clean":"2024-03-18T14:41:40.778816+0000","last_became_active":"2024-03-18T14:41:35.549894+0000","last_became_peered":"2024-03-18T14:41:35.549894+0000","last_unstale":"2024-03-18T14:41:40.778816+0000","last_undegraded":"2024-03-18T14:41:40.778816+0000","last_fullsized":"2024-03-18T14:41:40.778816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:45.045404+0000","last_change":"2024-03-18T14:41:34.536841+0000","last_active":"2024-03-18T14:41:45.045404+0000","last_peered":"2024-03-18T14:41:45.045404+0000","last_clean":"2024-03-18T14:41:45.045404+0000","last_became_active":"2024-03-18T14:41:34.536536+0000","last_became_peered":"2024-03-18T14:41:34.536536+0000","last_unstale":"2024-03-18T14:41:45.045404+0000","last_undegraded":"2024-03-18T14:41:45.045404+0000","last_fullsized":"2024-03-18T14:41:45.045404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779808+0000","last_change":"2024-03-18T14:41:39.582692+0000","last_active":"2024-03-18T14:41:40.779808+0000","last_peered":"2024-03-18T14:41:40.779808+0000","last_clean":"2024-03-18T14:41:40.779808+0000","last_became_active":"2024-03-18T14:41:39.582285+0000","last_became_peered":"2024-03-18T14:41:39.582285+0000","last_unstale":"2024-03-18T14:41:40.779808+0000","last_undegraded":"2024-03-18T14:41:40.779808+0000","last_fullsized":"2024-03-18T14:41:40.779808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559959+0000","last_change":"2024-03-18T14:41:35.547692+0000","last_active":"2024-03-18T14:41:40.559959+0000","last_peered":"2024-03-18T14:41:40.559959+0000","last_clean":"2024-03-18T14:41:40.559959+0000","last_became_active":"2024-03-18T14:41:35.547381+0000","last_became_peered":"2024-03-18T14:41:35.547381+0000","last_unstale":"2024-03-18T14:41:40.559959+0000","last_undegraded":"2024-03-18T14:41:40.559959+0000","last_fullsized":"2024-03-18T14:41:40.559959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:13.037877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779251+0000","last_change":"2024-03-18T14:41:34.529622+0000","last_active":"2024-03-18T14:41:40.779251+0000","last_peered":"2024-03-18T14:41:40.779251+0000","last_clean":"2024-03-18T14:41:40.779251+0000","last_became_active":"2024-03-18T14:41:34.528752+0000","last_became_peered":"2024-03-18T14:41:34.528752+0000","last_unstale":"2024-03-18T14:41:40.779251+0000","last_undegraded":"2024-03-18T14:41:40.779251+0000","last_fullsized":"2024-03-18T14:41:40.779251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560121+0000","last_change":"2024-03-18T14:41:39.585230+0000","last_active":"2024-03-18T14:41:40.560121+0000","last_peered":"2024-03-18T14:41:40.560121+0000","last_clean":"2024-03-18T14:41:40.560121+0000","last_became_active":"2024-03-18T14:41:39.584896+0000","last_became_peered":"2024-03-18T14:41:39.584896+0000","last_unstale":"2024-03-18T14:41:40.560121+0000","last_undegraded":"2024-03-18T14:41:40.560121+0000","last_fullsized":"2024-03-18T14:41:40.560121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:35.552508+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:35.552221+0000","last_became_peered":"2024-03-18T14:41:35.552221+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560164+0000","last_change":"2024-03-18T14:41:34.534226+0000","last_active":"2024-03-18T14:41:40.560164+0000","last_peered":"2024-03-18T14:41:40.560164+0000","last_clean":"2024-03-18T14:41:40.560164+0000","last_became_active":"2024-03-18T14:41:34.533977+0000","last_became_peered":"2024-03-18T14:41:34.533977+0000","last_unstale":"2024-03-18T14:41:40.560164+0000","last_undegraded":"2024-03-18T14:41:40.560164+0000","last_fullsized":"2024-03-18T14:41:40.560164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563415+0000","last_change":"2024-03-18T14:41:39.583657+0000","last_active":"2024-03-18T14:41:40.563415+0000","last_peered":"2024-03-18T14:41:40.563415+0000","last_clean":"2024-03-18T14:41:40.563415+0000","last_became_active":"2024-03-18T14:41:39.583280+0000","last_became_peered":"2024-03-18T14:41:39.583280+0000","last_unstale":"2024-03-18T14:41:40.563415+0000","last_undegraded":"2024-03-18T14:41:40.563415+0000","last_fullsized":"2024-03-18T14:41:40.563415+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560126+0000","last_change":"2024-03-18T14:41:35.546368+0000","last_active":"2024-03-18T14:41:40.560126+0000","last_peered":"2024-03-18T14:41:40.560126+0000","last_clean":"2024-03-18T14:41:40.560126+0000","last_became_active":"2024-03-18T14:41:35.546101+0000","last_became_peered":"2024-03-18T14:41:35.546101+0000","last_unstale":"2024-03-18T14:41:40.560126+0000","last_undegraded":"2024-03-18T14:41:40.560126+0000","last_fullsized":"2024-03-18T14:41:40.560126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779013+0000","last_change":"2024-03-18T14:41:34.535621+0000","last_active":"2024-03-18T14:41:40.779013+0000","last_peered":"2024-03-18T14:41:40.779013+0000","last_clean":"2024-03-18T14:41:40.779013+0000","last_became_active":"2024-03-18T14:41:34.535351+0000","last_became_peered":"2024-03-18T14:41:34.535351+0000","last_unstale":"2024-03-18T14:41:40.779013+0000","last_undegraded":"2024-03-18T14:41:40.779013+0000","last_fullsized":"2024-03-18T14:41:40.779013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563431+0000","last_change":"2024-03-18T14:41:39.602374+0000","last_active":"2024-03-18T14:41:40.563431+0000","last_peered":"2024-03-18T14:41:40.563431+0000","last_clean":"2024-03-18T14:41:40.563431+0000","last_became_active":"2024-03-18T14:41:39.600913+0000","last_became_peered":"2024-03-18T14:41:39.600913+0000","last_unstale":"2024-03-18T14:41:40.563431+0000","last_undegraded":"2024-03-18T14:41:40.563431+0000","last_fullsized":"2024-03-18T14:41:40.563431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778290+0000","last_change":"2024-03-18T14:41:39.602659+0000","last_active":"2024-03-18T14:41:40.778290+0000","last_peered":"2024-03-18T14:41:40.778290+0000","last_clean":"2024-03-18T14:41:40.778290+0000","last_became_active":"2024-03-18T14:41:39.602286+0000","last_became_peered":"2024-03-18T14:41:39.602286+0000","last_unstale":"2024-03-18T14:41:40.778290+0000","last_undegraded":"2024-03-18T14:41:40.778290+0000","last_fullsized":"2024-03-18T14:41:40.778290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778038+0000","last_change":"2024-03-18T14:41:35.545020+0000","last_active":"2024-03-18T14:41:40.778038+0000","last_peered":"2024-03-18T14:41:40.778038+0000","last_clean":"2024-03-18T14:41:40.778038+0000","last_became_active":"2024-03-18T14:41:35.544697+0000","last_became_peered":"2024-03-18T14:41:35.544697+0000","last_unstale":"2024-03-18T14:41:40.778038+0000","last_undegraded":"2024-03-18T14:41:40.778038+0000","last_fullsized":"2024-03-18T14:41:40.778038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559154+0000","last_change":"2024-03-18T14:41:34.527624+0000","last_active":"2024-03-18T14:41:40.559154+0000","last_peered":"2024-03-18T14:41:40.559154+0000","last_clean":"2024-03-18T14:41:40.559154+0000","last_became_active":"2024-03-18T14:41:34.527291+0000","last_became_peered":"2024-03-18T14:41:34.527291+0000","last_unstale":"2024-03-18T14:41:40.559154+0000","last_undegraded":"2024-03-18T14:41:40.559154+0000","last_fullsized":"2024-03-18T14:41:40.559154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779046+0000","last_change":"2024-03-18T14:41:39.575946+0000","last_active":"2024-03-18T14:41:40.779046+0000","last_peered":"2024-03-18T14:41:40.779046+0000","last_clean":"2024-03-18T14:41:40.779046+0000","last_became_active":"2024-03-18T14:41:39.575618+0000","last_became_peered":"2024-03-18T14:41:39.575618+0000","last_unstale":"2024-03-18T14:41:40.779046+0000","last_undegraded":"2024-03-18T14:41:40.779046+0000","last_fullsized":"2024-03-18T14:41:40.779046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563258+0000","last_change":"2024-03-18T14:41:35.548573+0000","last_active":"2024-03-18T14:41:40.563258+0000","last_peered":"2024-03-18T14:41:40.563258+0000","last_clean":"2024-03-18T14:41:40.563258+0000","last_became_active":"2024-03-18T14:41:35.548284+0000","last_became_peered":"2024-03-18T14:41:35.548284+0000","last_unstale":"2024-03-18T14:41:40.563258+0000","last_undegraded":"2024-03-18T14:41:40.563258+0000","last_fullsized":"2024-03-18T14:41:40.563258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563616+0000","last_change":"2024-03-18T14:41:34.536158+0000","last_active":"2024-03-18T14:41:40.563616+0000","last_peered":"2024-03-18T14:41:40.563616+0000","last_clean":"2024-03-18T14:41:40.563616+0000","last_became_active":"2024-03-18T14:41:34.535879+0000","last_became_peered":"2024-03-18T14:41:34.535879+0000","last_unstale":"2024-03-18T14:41:40.563616+0000","last_undegraded":"2024-03-18T14:41:40.563616+0000","last_fullsized":"2024-03-18T14:41:40.563616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778365+0000","last_change":"2024-03-18T14:41:34.528073+0000","last_active":"2024-03-18T14:41:40.778365+0000","last_peered":"2024-03-18T14:41:40.778365+0000","last_clean":"2024-03-18T14:41:40.778365+0000","last_became_active":"2024-03-18T14:41:34.527245+0000","last_became_peered":"2024-03-18T14:41:34.527245+0000","last_unstale":"2024-03-18T14:41:40.778365+0000","last_undegraded":"2024-03-18T14:41:40.778365+0000","last_fullsized":"2024-03-18T14:41:40.778365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779106+0000","last_change":"2024-03-18T14:41:35.560334+0000","last_active":"2024-03-18T14:41:40.779106+0000","last_peered":"2024-03-18T14:41:40.779106+0000","last_clean":"2024-03-18T14:41:40.779106+0000","last_became_active":"2024-03-18T14:41:35.559938+0000","last_became_peered":"2024-03-18T14:41:35.559938+0000","last_unstale":"2024-03-18T14:41:40.779106+0000","last_undegraded":"2024-03-18T14:41:40.779106+0000","last_fullsized":"2024-03-18T14:41:40.779106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780010+0000","last_change":"2024-03-18T14:41:39.578410+0000","last_active":"2024-03-18T14:41:40.780010+0000","last_peered":"2024-03-18T14:41:40.780010+0000","last_clean":"2024-03-18T14:41:40.780010+0000","last_became_active":"2024-03-18T14:41:39.577919+0000","last_became_peered":"2024-03-18T14:41:39.577919+0000","last_unstale":"2024-03-18T14:41:40.780010+0000","last_undegraded":"2024-03-18T14:41:40.780010+0000","last_fullsized":"2024-03-18T14:41:40.780010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:59.569962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563940+0000","last_change":"2024-03-18T14:41:34.535233+0000","last_active":"2024-03-18T14:41:40.563940+0000","last_peered":"2024-03-18T14:41:40.563940+0000","last_clean":"2024-03-18T14:41:40.563940+0000","last_became_active":"2024-03-18T14:41:34.534879+0000","last_became_peered":"2024-03-18T14:41:34.534879+0000","last_unstale":"2024-03-18T14:41:40.563940+0000","last_undegraded":"2024-03-18T14:41:40.563940+0000","last_fullsized":"2024-03-18T14:41:40.563940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559603+0000","last_change":"2024-03-18T14:41:35.535546+0000","last_active":"2024-03-18T14:41:40.559603+0000","last_peered":"2024-03-18T14:41:40.559603+0000","last_clean":"2024-03-18T14:41:40.559603+0000","last_became_active":"2024-03-18T14:41:35.535269+0000","last_became_peered":"2024-03-18T14:41:35.535269+0000","last_unstale":"2024-03-18T14:41:40.559603+0000","last_undegraded":"2024-03-18T14:41:40.559603+0000","last_fullsized":"2024-03-18T14:41:40.559603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:39.583244+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:39.582882+0000","last_became_peered":"2024-03-18T14:41:39.582882+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:10:48.564938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777919+0000","last_change":"2024-03-18T14:41:35.554708+0000","last_active":"2024-03-18T14:41:40.777919+0000","last_peered":"2024-03-18T14:41:40.777919+0000","last_clean":"2024-03-18T14:41:40.777919+0000","last_became_active":"2024-03-18T14:41:35.554411+0000","last_became_peered":"2024-03-18T14:41:35.554411+0000","last_unstale":"2024-03-18T14:41:40.777919+0000","last_undegraded":"2024-03-18T14:41:40.777919+0000","last_fullsized":"2024-03-18T14:41:40.777919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777933+0000","last_change":"2024-03-18T14:41:35.543401+0000","last_active":"2024-03-18T14:41:40.777933+0000","last_peered":"2024-03-18T14:41:40.777933+0000","last_clean":"2024-03-18T14:41:40.777933+0000","last_became_active":"2024-03-18T14:41:35.543034+0000","last_became_peered":"2024-03-18T14:41:35.543034+0000","last_unstale":"2024-03-18T14:41:40.777933+0000","last_undegraded":"2024-03-18T14:41:40.777933+0000","last_fullsized":"2024-03-18T14:41:40.777933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559616+0000","last_change":"2024-03-18T14:41:35.551162+0000","last_active":"2024-03-18T14:41:40.559616+0000","last_peered":"2024-03-18T14:41:40.559616+0000","last_clean":"2024-03-18T14:41:40.559616+0000","last_became_active":"2024-03-18T14:41:35.550839+0000","last_became_peered":"2024-03-18T14:41:35.550839+0000","last_unstale":"2024-03-18T14:41:40.559616+0000","last_undegraded":"2024-03-18T14:41:40.559616+0000","last_fullsized":"2024-03-18T14:41:40.559616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779023+0000","last_change":"2024-03-18T14:41:35.554787+0000","last_active":"2024-03-18T14:41:40.779023+0000","last_peered":"2024-03-18T14:41:40.779023+0000","last_clean":"2024-03-18T14:41:40.779023+0000","last_became_active":"2024-03-18T14:41:35.554484+0000","last_became_peered":"2024-03-18T14:41:35.554484+0000","last_unstale":"2024-03-18T14:41:40.779023+0000","last_undegraded":"2024-03-18T14:41:40.779023+0000","last_fullsized":"2024-03-18T14:41:40.779023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564088+0000","last_change":"2024-03-18T14:41:35.544958+0000","last_active":"2024-03-18T14:41:40.564088+0000","last_peered":"2024-03-18T14:41:40.564088+0000","last_clean":"2024-03-18T14:41:40.564088+0000","last_became_active":"2024-03-18T14:41:35.544600+0000","last_became_peered":"2024-03-18T14:41:35.544600+0000","last_unstale":"2024-03-18T14:41:40.564088+0000","last_undegraded":"2024-03-18T14:41:40.564088+0000","last_fullsized":"2024-03-18T14:41:40.564088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778046+0000","last_change":"2024-03-18T14:41:35.549309+0000","last_active":"2024-03-18T14:41:40.778046+0000","last_peered":"2024-03-18T14:41:40.778046+0000","last_clean":"2024-03-18T14:41:40.778046+0000","last_became_active":"2024-03-18T14:41:35.549039+0000","last_became_peered":"2024-03-18T14:41:35.549039+0000","last_unstale":"2024-03-18T14:41:40.778046+0000","last_undegraded":"2024-03-18T14:41:40.778046+0000","last_fullsized":"2024-03-18T14:41:40.778046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778908+0000","last_change":"2024-03-18T14:41:35.563387+0000","last_active":"2024-03-18T14:41:40.778908+0000","last_peered":"2024-03-18T14:41:40.778908+0000","last_clean":"2024-03-18T14:41:40.778908+0000","last_became_active":"2024-03-18T14:41:35.562621+0000","last_became_peered":"2024-03-18T14:41:35.562621+0000","last_unstale":"2024-03-18T14:41:40.778908+0000","last_undegraded":"2024-03-18T14:41:40.778908+0000","last_fullsized":"2024-03-18T14:41:40.778908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564514+0000","last_change":"2024-03-18T14:41:35.549988+0000","last_active":"2024-03-18T14:41:40.564514+0000","last_peered":"2024-03-18T14:41:40.564514+0000","last_clean":"2024-03-18T14:41:40.564514+0000","last_became_active":"2024-03-18T14:41:35.549713+0000","last_became_peered":"2024-03-18T14:41:35.549713+0000","last_unstale":"2024-03-18T14:41:40.564514+0000","last_undegraded":"2024-03-18T14:41:40.564514+0000","last_fullsized":"2024-03-18T14:41:40.564514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559089+0000","last_change":"2024-03-18T14:41:35.540097+0000","last_active":"2024-03-18T14:41:40.559089+0000","last_peered":"2024-03-18T14:41:40.559089+0000","last_clean":"2024-03-18T14:41:40.559089+0000","last_became_active":"2024-03-18T14:41:35.539823+0000","last_became_peered":"2024-03-18T14:41:35.539823+0000","last_unstale":"2024-03-18T14:41:40.559089+0000","last_undegraded":"2024-03-18T14:41:40.559089+0000","last_fullsized":"2024-03-18T14:41:40.559089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563495+0000","last_change":"2024-03-18T14:41:35.544547+0000","last_active":"2024-03-18T14:41:40.563495+0000","last_peered":"2024-03-18T14:41:40.563495+0000","last_clean":"2024-03-18T14:41:40.563495+0000","last_became_active":"2024-03-18T14:41:35.544026+0000","last_became_peered":"2024-03-18T14:41:35.544026+0000","last_unstale":"2024-03-18T14:41:40.563495+0000","last_undegraded":"2024-03-18T14:41:40.563495+0000","last_fullsized":"2024-03-18T14:41:40.563495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559538+0000","last_change":"2024-03-18T14:41:35.545773+0000","last_active":"2024-03-18T14:41:40.559538+0000","last_peered":"2024-03-18T14:41:40.559538+0000","last_clean":"2024-03-18T14:41:40.559538+0000","last_became_active":"2024-03-18T14:41:35.545449+0000","last_became_peered":"2024-03-18T14:41:35.545449+0000","last_unstale":"2024-03-18T14:41:40.559538+0000","last_undegraded":"2024-03-18T14:41:40.559538+0000","last_fullsized":"2024-03-18T14:41:40.559538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778786+0000","last_change":"2024-03-18T14:41:35.563339+0000","last_active":"2024-03-18T14:41:40.778786+0000","last_peered":"2024-03-18T14:41:40.778786+0000","last_clean":"2024-03-18T14:41:40.778786+0000","last_became_active":"2024-03-18T14:41:35.563117+0000","last_became_peered":"2024-03-18T14:41:35.563117+0000","last_unstale":"2024-03-18T14:41:40.778786+0000","last_undegraded":"2024-03-18T14:41:40.778786+0000","last_fullsized":"2024-03-18T14:41:40.778786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778522+0000","last_change":"2024-03-18T14:41:35.559014+0000","last_active":"2024-03-18T14:41:40.778522+0000","last_peered":"2024-03-18T14:41:40.778522+0000","last_clean":"2024-03-18T14:41:40.778522+0000","last_became_active":"2024-03-18T14:41:35.558349+0000","last_became_peered":"2024-03-18T14:41:35.558349+0000","last_unstale":"2024-03-18T14:41:40.778522+0000","last_undegraded":"2024-03-18T14:41:40.778522+0000","last_fullsized":"2024-03-18T14:41:40.778522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778971+0000","last_change":"2024-03-18T14:41:35.545841+0000","last_active":"2024-03-18T14:41:40.778971+0000","last_peered":"2024-03-18T14:41:40.778971+0000","last_clean":"2024-03-18T14:41:40.778971+0000","last_became_active":"2024-03-18T14:41:35.545543+0000","last_became_peered":"2024-03-18T14:41:35.545543+0000","last_unstale":"2024-03-18T14:41:40.778971+0000","last_undegraded":"2024-03-18T14:41:40.778971+0000","last_fullsized":"2024-03-18T14:41:40.778971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778782+0000","last_change":"2024-03-18T14:41:35.551018+0000","last_active":"2024-03-18T14:41:40.778782+0000","last_peered":"2024-03-18T14:41:40.778782+0000","last_clean":"2024-03-18T14:41:40.778782+0000","last_became_active":"2024-03-18T14:41:35.550737+0000","last_became_peered":"2024-03-18T14:41:35.550737+0000","last_unstale":"2024-03-18T14:41:40.778782+0000","last_undegraded":"2024-03-18T14:41:40.778782+0000","last_fullsized":"2024-03-18T14:41:40.778782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560481+0000","last_change":"2024-03-18T14:41:35.547133+0000","last_active":"2024-03-18T14:41:40.560481+0000","last_peered":"2024-03-18T14:41:40.560481+0000","last_clean":"2024-03-18T14:41:40.560481+0000","last_became_active":"2024-03-18T14:41:35.546778+0000","last_became_peered":"2024-03-18T14:41:35.546778+0000","last_unstale":"2024-03-18T14:41:40.560481+0000","last_undegraded":"2024-03-18T14:41:40.560481+0000","last_fullsized":"2024-03-18T14:41:40.560481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559067+0000","last_change":"2024-03-18T14:41:35.550366+0000","last_active":"2024-03-18T14:41:40.559067+0000","last_peered":"2024-03-18T14:41:40.559067+0000","last_clean":"2024-03-18T14:41:40.559067+0000","last_became_active":"2024-03-18T14:41:35.550111+0000","last_became_peered":"2024-03-18T14:41:35.550111+0000","last_unstale":"2024-03-18T14:41:40.559067+0000","last_undegraded":"2024-03-18T14:41:40.559067+0000","last_fullsized":"2024-03-18T14:41:40.559067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778058+0000","last_change":"2024-03-18T14:41:35.536085+0000","last_active":"2024-03-18T14:41:40.778058+0000","last_peered":"2024-03-18T14:41:40.778058+0000","last_clean":"2024-03-18T14:41:40.778058+0000","last_became_active":"2024-03-18T14:41:35.535769+0000","last_became_peered":"2024-03-18T14:41:35.535769+0000","last_unstale":"2024-03-18T14:41:40.778058+0000","last_undegraded":"2024-03-18T14:41:40.778058+0000","last_fullsized":"2024-03-18T14:41:40.778058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778571+0000","last_change":"2024-03-18T14:41:35.550291+0000","last_active":"2024-03-18T14:41:40.778571+0000","last_peered":"2024-03-18T14:41:40.778571+0000","last_clean":"2024-03-18T14:41:40.778571+0000","last_became_active":"2024-03-18T14:41:35.549994+0000","last_became_peered":"2024-03-18T14:41:35.549994+0000","last_unstale":"2024-03-18T14:41:40.778571+0000","last_undegraded":"2024-03-18T14:41:40.778571+0000","last_fullsized":"2024-03-18T14:41:40.778571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:20.446494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563821+0000","last_change":"2024-03-18T14:41:35.551464+0000","last_active":"2024-03-18T14:41:40.563821+0000","last_peered":"2024-03-18T14:41:40.563821+0000","last_clean":"2024-03-18T14:41:40.563821+0000","last_became_active":"2024-03-18T14:41:35.551200+0000","last_became_peered":"2024-03-18T14:41:35.551200+0000","last_unstale":"2024-03-18T14:41:40.563821+0000","last_undegraded":"2024-03-18T14:41:40.563821+0000","last_fullsized":"2024-03-18T14:41:40.563821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559869+0000","last_change":"2024-03-18T14:41:35.558583+0000","last_active":"2024-03-18T14:41:40.559869+0000","last_peered":"2024-03-18T14:41:40.559869+0000","last_clean":"2024-03-18T14:41:40.559869+0000","last_became_active":"2024-03-18T14:41:35.557388+0000","last_became_peered":"2024-03-18T14:41:35.557388+0000","last_unstale":"2024-03-18T14:41:40.559869+0000","last_undegraded":"2024-03-18T14:41:40.559869+0000","last_fullsized":"2024-03-18T14:41:40.559869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778685+0000","last_change":"2024-03-18T14:41:35.556362+0000","last_active":"2024-03-18T14:41:40.778685+0000","last_peered":"2024-03-18T14:41:40.778685+0000","last_clean":"2024-03-18T14:41:40.778685+0000","last_became_active":"2024-03-18T14:41:35.556073+0000","last_became_peered":"2024-03-18T14:41:35.556073+0000","last_unstale":"2024-03-18T14:41:40.778685+0000","last_undegraded":"2024-03-18T14:41:40.778685+0000","last_fullsized":"2024-03-18T14:41:40.778685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778827+0000","last_change":"2024-03-18T14:41:35.553776+0000","last_active":"2024-03-18T14:41:40.778827+0000","last_peered":"2024-03-18T14:41:40.778827+0000","last_clean":"2024-03-18T14:41:40.778827+0000","last_became_active":"2024-03-18T14:41:35.553413+0000","last_became_peered":"2024-03-18T14:41:35.553413+0000","last_unstale":"2024-03-18T14:41:40.778827+0000","last_undegraded":"2024-03-18T14:41:40.778827+0000","last_fullsized":"2024-03-18T14:41:40.778827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778851+0000","last_change":"2024-03-18T14:41:35.556025+0000","last_active":"2024-03-18T14:41:40.778851+0000","last_peered":"2024-03-18T14:41:40.778851+0000","last_clean":"2024-03-18T14:41:40.778851+0000","last_became_active":"2024-03-18T14:41:35.555322+0000","last_became_peered":"2024-03-18T14:41:35.555322+0000","last_unstale":"2024-03-18T14:41:40.778851+0000","last_undegraded":"2024-03-18T14:41:40.778851+0000","last_fullsized":"2024-03-18T14:41:40.778851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563141+0000","last_change":"2024-03-18T14:41:35.545290+0000","last_active":"2024-03-18T14:41:40.563141+0000","last_peered":"2024-03-18T14:41:40.563141+0000","last_clean":"2024-03-18T14:41:40.563141+0000","last_became_active":"2024-03-18T14:41:35.543962+0000","last_became_peered":"2024-03-18T14:41:35.543962+0000","last_unstale":"2024-03-18T14:41:40.563141+0000","last_undegraded":"2024-03-18T14:41:40.563141+0000","last_fullsized":"2024-03-18T14:41:40.563141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778288+0000","last_change":"2024-03-18T14:41:35.556987+0000","last_active":"2024-03-18T14:41:40.778288+0000","last_peered":"2024-03-18T14:41:40.778288+0000","last_clean":"2024-03-18T14:41:40.778288+0000","last_became_active":"2024-03-18T14:41:35.556713+0000","last_became_peered":"2024-03-18T14:41:35.556713+0000","last_unstale":"2024-03-18T14:41:40.778288+0000","last_undegraded":"2024-03-18T14:41:40.778288+0000","last_fullsized":"2024-03-18T14:41:40.778288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559461+0000","last_change":"2024-03-18T14:41:35.548724+0000","last_active":"2024-03-18T14:41:40.559461+0000","last_peered":"2024-03-18T14:41:40.559461+0000","last_clean":"2024-03-18T14:41:40.559461+0000","last_became_active":"2024-03-18T14:41:35.547693+0000","last_became_peered":"2024-03-18T14:41:35.547693+0000","last_unstale":"2024-03-18T14:41:40.559461+0000","last_undegraded":"2024-03-18T14:41:40.559461+0000","last_fullsized":"2024-03-18T14:41:40.559461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:35.544193+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:35.543909+0000","last_became_peered":"2024-03-18T14:41:35.543909+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559445+0000","last_change":"2024-03-18T14:41:35.549929+0000","last_active":"2024-03-18T14:41:40.559445+0000","last_peered":"2024-03-18T14:41:40.559445+0000","last_clean":"2024-03-18T14:41:40.559445+0000","last_became_active":"2024-03-18T14:41:35.547332+0000","last_became_peered":"2024-03-18T14:41:35.547332+0000","last_unstale":"2024-03-18T14:41:40.559445+0000","last_undegraded":"2024-03-18T14:41:40.559445+0000","last_fullsized":"2024-03-18T14:41:40.559445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779000+0000","last_change":"2024-03-18T14:41:35.550445+0000","last_active":"2024-03-18T14:41:40.779000+0000","last_peered":"2024-03-18T14:41:40.779000+0000","last_clean":"2024-03-18T14:41:40.779000+0000","last_became_active":"2024-03-18T14:41:35.550120+0000","last_became_peered":"2024-03-18T14:41:35.550120+0000","last_unstale":"2024-03-18T14:41:40.779000+0000","last_undegraded":"2024-03-18T14:41:40.779000+0000","last_fullsized":"2024-03-18T14:41:40.779000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778964+0000","last_change":"2024-03-18T14:41:35.548653+0000","last_active":"2024-03-18T14:41:40.778964+0000","last_peered":"2024-03-18T14:41:40.778964+0000","last_clean":"2024-03-18T14:41:40.778964+0000","last_became_active":"2024-03-18T14:41:35.548305+0000","last_became_peered":"2024-03-18T14:41:35.548305+0000","last_unstale":"2024-03-18T14:41:40.778964+0000","last_undegraded":"2024-03-18T14:41:40.778964+0000","last_fullsized":"2024-03-18T14:41:40.778964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778662+0000","last_change":"2024-03-18T14:41:35.560778+0000","last_active":"2024-03-18T14:41:40.778662+0000","last_peered":"2024-03-18T14:41:40.778662+0000","last_clean":"2024-03-18T14:41:40.778662+0000","last_became_active":"2024-03-18T14:41:35.560478+0000","last_became_peered":"2024-03-18T14:41:35.560478+0000","last_unstale":"2024-03-18T14:41:40.778662+0000","last_undegraded":"2024-03-18T14:41:40.778662+0000","last_fullsized":"2024-03-18T14:41:40.778662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:42:03.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:42:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.164+0000 7f13391b9640 1 Processor -- start 2024-03-18T14:42:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.164+0000 7f13391b9640 1 -- start start 2024-03-18T14:42:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f13391b9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f133415bab0 0x7f133415be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f13391b9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f133406e9c0 0x7f13341602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f13391b9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13340b2dd0 con 0x7f133406dbb0 2024-03-18T14:42:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f13391b9640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1334160820 con 0x7f133415bab0 2024-03-18T14:42:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f13391b9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f13340b30b0 con 0x7f133406e9c0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f1332d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f133415bab0 0x7f133415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f1332d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f133415bab0 0x7f133415be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56758/0 (socket says 172.21.15.6:56758) 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f1332d9d640 1 -- 172.21.15.6:0/3006718911 learned_addr learned my addr 172.21.15.6:0/3006718911 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f133359e640 1 --2- 172.21.15.6:0/3006718911 >> v2:172.21.15.92:3300/0 conn(0x7f133406e9c0 0x7f13341602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f1332d9d640 1 -- 172.21.15.6:0/3006718911 >> v2:172.21.15.92:3300/0 conn(0x7f133406e9c0 msgr2=0x7f13341602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f1332d9d640 1 --2- 172.21.15.6:0/3006718911 >> v2:172.21.15.92:3300/0 conn(0x7f133406e9c0 0x7f13341602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f1332d9d640 1 -- 172.21.15.6:0/3006718911 >> v1:172.21.15.92:6789/0 conn(0x7f133406dbb0 legacy=0x7f133406e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.165+0000 7f1332d9d640 1 -- 172.21.15.6:0/3006718911 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13340b3390 con 0x7f133415bab0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f1332d9d640 1 --2- 172.21.15.6:0/3006718911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133415bab0 0x7f133415be90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f133415f7e0 tx=0x7f132c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987f33152beb2caf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f1331d9b640 1 -- 172.21.15.6:0/3006718911 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c034070 con 0x7f133415bab0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f1331d9b640 1 -- 172.21.15.6:0/3006718911 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f132c030860 con 0x7f133415bab0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f1331d9b640 1 -- 172.21.15.6:0/3006718911 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c030c30 con 0x7f133415bab0 2024-03-18T14:42:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f13391b9640 1 -- 172.21.15.6:0/3006718911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133415bab0 msgr2=0x7f133415be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f13391b9640 1 --2- 172.21.15.6:0/3006718911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133415bab0 0x7f133415be90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f133415f7e0 tx=0x7f132c02b9a0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f13391b9640 1 -- 172.21.15.6:0/3006718911 shutdown_connections 2024-03-18T14:42:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f13391b9640 1 --2- 172.21.15.6:0/3006718911 >> v2:172.21.15.92:3300/0 conn(0x7f133406e9c0 0x7f13341602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f13391b9640 1 --2- 172.21.15.6:0/3006718911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133415bab0 0x7f133415be90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.166+0000 7f13391b9640 1 -- 172.21.15.6:0/3006718911 >> 172.21.15.6:0/3006718911 conn(0x7f13340b1470 msgr2=0x7f13340b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.167+0000 7f13391b9640 1 -- 172.21.15.6:0/3006718911 shutdown_connections 2024-03-18T14:42:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.167+0000 7f13391b9640 1 -- 172.21.15.6:0/3006718911 wait complete. 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.168+0000 7f13391b9640 1 Processor -- start 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.168+0000 7f13391b9640 1 -- start start 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133406dbb0 0x7f13340cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133406e9c0 0x7f13340cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133415bab0 0x7f1334146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f13340b3d10 con 0x7f133406e9c0 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13340b3b90 con 0x7f133406dbb0 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f13340b3e90 con 0x7f133415bab0 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133406e9c0 0x7f13340cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133406e9c0 0x7f13340cf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56768/0 (socket says 172.21.15.6:56768) 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 -- 172.21.15.6:0/2018637243 learned_addr learned my addr 172.21.15.6:0/2018637243 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f1332d9d640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133406dbb0 0x7f13340cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133359e640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133415bab0 0x7f1334146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 -- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133415bab0 msgr2=0x7f1334146ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133415bab0 0x7f1334146ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 -- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133406dbb0 msgr2=0x7f13340cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133406dbb0 0x7f13340cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 -- 172.21.15.6:0/2018637243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13340c50b0 con 0x7f133406e9c0 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f133259c640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133406e9c0 0x7f13340cf9c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f131c00c400 tx=0x7f131c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f132b7fe640 1 -- 172.21.15.6:0/2018637243 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c0052d0 con 0x7f133406e9c0 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f132b7fe640 1 -- 172.21.15.6:0/2018637243 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c005490 con 0x7f133406e9c0 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f132b7fe640 1 -- 172.21.15.6:0/2018637243 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c007260 con 0x7f133406e9c0 2024-03-18T14:42:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13340c5390 con 0x7f133406e9c0 2024-03-18T14:42:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.169+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13340c5c30 con 0x7f133406e9c0 2024-03-18T14:42:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.170+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13340c5560 con 0x7f133406e9c0 2024-03-18T14:42:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.171+0000 7f132b7fe640 1 -- 172.21.15.6:0/2018637243 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f131c007400 con 0x7f133406e9c0 2024-03-18T14:42:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.171+0000 7f132b7fe640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f131406bbd0 0x7f131406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.171+0000 7f132b7fe640 1 -- 172.21.15.6:0/2018637243 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f131c08a270 con 0x7f133406e9c0 2024-03-18T14:42:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.171+0000 7f1332d9d640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f131406bbd0 0x7f131406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.172+0000 7f1332d9d640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f131406bbd0 0x7f131406e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f132c036000 tx=0x7f132c03b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.176+0000 7f132b7fe640 1 -- 172.21.15.6:0/2018637243 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f131c0583d0 con 0x7f133406e9c0 2024-03-18T14:42:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.357+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1334073310 con 0x7f131406bbd0 2024-03-18T14:42:03.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.365+0000 7f132b7fe640 1 -- 172.21.15.6:0/2018637243 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f1334073310 con 0x7f131406bbd0 2024-03-18T14:42:03.370 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:03.371 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:42:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f131406bbd0 msgr2=0x7f131406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f131406bbd0 0x7f131406e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f132c036000 tx=0x7f132c03b040 comp rx=0 tx=0).stop 2024-03-18T14:42:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133406e9c0 msgr2=0x7f13340cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133406e9c0 0x7f13340cf9c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f131c00c400 tx=0x7f131c00c8c0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 shutdown_connections 2024-03-18T14:42:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f131406bbd0 0x7f131406e090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133415bab0 0x7f1334146ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f133406e9c0 0x7f13340cf9c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 --2- 172.21.15.6:0/2018637243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133406dbb0 0x7f13340cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 >> 172.21.15.6:0/2018637243 conn(0x7f13340b1470 msgr2=0x7f13340cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 shutdown_connections 2024-03-18T14:42:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.368+0000 7f13391b9640 1 -- 172.21.15.6:0/2018637243 wait complete. 2024-03-18T14:42:03.381 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-18T14:42:01.385243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002586"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779658+0000","last_change":"2024-03-18T14:41:34.533975+0000","last_active":"2024-03-18T14:41:40.779658+0000","last_peered":"2024-03-18T14:41:40.779658+0000","last_clean":"2024-03-18T14:41:40.779658+0000","last_became_active":"2024-03-18T14:41:34.533697+0000","last_became_peered":"2024-03-18T14:41:34.533697+0000","last_unstale":"2024-03-18T14:41:40.779658+0000","last_undegraded":"2024-03-18T14:41:40.779658+0000","last_fullsized":"2024-03-18T14:41:40.779658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:35.555471+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:35.555193+0000","last_became_peered":"2024-03-18T14:41:35.555193+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563408+0000","last_change":"2024-03-18T14:41:39.604430+0000","last_active":"2024-03-18T14:41:40.563408+0000","last_peered":"2024-03-18T14:41:40.563408+0000","last_clean":"2024-03-18T14:41:40.563408+0000","last_became_active":"2024-03-18T14:41:39.603958+0000","last_became_peered":"2024-03-18T14:41:39.603958+0000","last_unstale":"2024-03-18T14:41:40.563408+0000","last_undegraded":"2024-03-18T14:41:40.563408+0000","last_fullsized":"2024-03-18T14:41:40.563408+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:17.770515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564369+0000","last_change":"2024-03-18T14:41:34.535172+0000","last_active":"2024-03-18T14:41:40.564369+0000","last_peered":"2024-03-18T14:41:40.564369+0000","last_clean":"2024-03-18T14:41:40.564369+0000","last_became_active":"2024-03-18T14:41:34.534849+0000","last_became_peered":"2024-03-18T14:41:34.534849+0000","last_unstale":"2024-03-18T14:41:40.564369+0000","last_undegraded":"2024-03-18T14:41:40.564369+0000","last_fullsized":"2024-03-18T14:41:40.564369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779347+0000","last_change":"2024-03-18T14:41:35.553407+0000","last_active":"2024-03-18T14:41:40.779347+0000","last_peered":"2024-03-18T14:41:40.779347+0000","last_clean":"2024-03-18T14:41:40.779347+0000","last_became_active":"2024-03-18T14:41:35.553091+0000","last_became_peered":"2024-03-18T14:41:35.553091+0000","last_unstale":"2024-03-18T14:41:40.779347+0000","last_undegraded":"2024-03-18T14:41:40.779347+0000","last_fullsized":"2024-03-18T14:41:40.779347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559102+0000","last_change":"2024-03-18T14:41:39.579025+0000","last_active":"2024-03-18T14:41:40.559102+0000","last_peered":"2024-03-18T14:41:40.559102+0000","last_clean":"2024-03-18T14:41:40.559102+0000","last_became_active":"2024-03-18T14:41:39.578650+0000","last_became_peered":"2024-03-18T14:41:39.578650+0000","last_unstale":"2024-03-18T14:41:40.559102+0000","last_undegraded":"2024-03-18T14:41:40.559102+0000","last_fullsized":"2024-03-18T14:41:40.559102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:53.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778929+0000","last_change":"2024-03-18T14:41:34.536627+0000","last_active":"2024-03-18T14:41:40.778929+0000","last_peered":"2024-03-18T14:41:40.778929+0000","last_clean":"2024-03-18T14:41:40.778929+0000","last_became_active":"2024-03-18T14:41:34.536136+0000","last_became_peered":"2024-03-18T14:41:34.536136+0000","last_unstale":"2024-03-18T14:41:40.778929+0000","last_undegraded":"2024-03-18T14:41:40.778929+0000","last_fullsized":"2024-03-18T14:41:40.778929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:12.036369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559536+0000","last_change":"2024-03-18T14:41:35.544918+0000","last_active":"2024-03-18T14:41:40.559536+0000","last_peered":"2024-03-18T14:41:40.559536+0000","last_clean":"2024-03-18T14:41:40.559536+0000","last_became_active":"2024-03-18T14:41:35.544612+0000","last_became_peered":"2024-03-18T14:41:35.544612+0000","last_unstale":"2024-03-18T14:41:40.559536+0000","last_undegraded":"2024-03-18T14:41:40.559536+0000","last_fullsized":"2024-03-18T14:41:40.559536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560251+0000","last_change":"2024-03-18T14:41:39.580665+0000","last_active":"2024-03-18T14:41:40.560251+0000","last_peered":"2024-03-18T14:41:40.560251+0000","last_clean":"2024-03-18T14:41:40.560251+0000","last_became_active":"2024-03-18T14:41:39.580231+0000","last_became_peered":"2024-03-18T14:41:39.580231+0000","last_unstale":"2024-03-18T14:41:40.560251+0000","last_undegraded":"2024-03-18T14:41:40.560251+0000","last_fullsized":"2024-03-18T14:41:40.560251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778491+0000","last_change":"2024-03-18T14:41:34.531008+0000","last_active":"2024-03-18T14:41:40.778491+0000","last_peered":"2024-03-18T14:41:40.778491+0000","last_clean":"2024-03-18T14:41:40.778491+0000","last_became_active":"2024-03-18T14:41:34.530587+0000","last_became_peered":"2024-03-18T14:41:34.530587+0000","last_unstale":"2024-03-18T14:41:40.778491+0000","last_undegraded":"2024-03-18T14:41:40.778491+0000","last_fullsized":"2024-03-18T14:41:40.778491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560259+0000","last_change":"2024-03-18T14:41:35.542756+0000","last_active":"2024-03-18T14:41:40.560259+0000","last_peered":"2024-03-18T14:41:40.560259+0000","last_clean":"2024-03-18T14:41:40.560259+0000","last_became_active":"2024-03-18T14:41:35.542486+0000","last_became_peered":"2024-03-18T14:41:35.542486+0000","last_unstale":"2024-03-18T14:41:40.560259+0000","last_undegraded":"2024-03-18T14:41:40.560259+0000","last_fullsized":"2024-03-18T14:41:40.560259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559437+0000","last_change":"2024-03-18T14:41:39.578229+0000","last_active":"2024-03-18T14:41:40.559437+0000","last_peered":"2024-03-18T14:41:40.559437+0000","last_clean":"2024-03-18T14:41:40.559437+0000","last_became_active":"2024-03-18T14:41:39.577812+0000","last_became_peered":"2024-03-18T14:41:39.577812+0000","last_unstale":"2024-03-18T14:41:40.559437+0000","last_undegraded":"2024-03-18T14:41:40.559437+0000","last_fullsized":"2024-03-18T14:41:40.559437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:50.123586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778041+0000","last_change":"2024-03-18T14:41:35.549686+0000","last_active":"2024-03-18T14:41:40.778041+0000","last_peered":"2024-03-18T14:41:40.778041+0000","last_clean":"2024-03-18T14:41:40.778041+0000","last_became_active":"2024-03-18T14:41:35.549240+0000","last_became_peered":"2024-03-18T14:41:35.549240+0000","last_unstale":"2024-03-18T14:41:40.778041+0000","last_undegraded":"2024-03-18T14:41:40.778041+0000","last_fullsized":"2024-03-18T14:41:40.778041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778484+0000","last_change":"2024-03-18T14:41:34.530798+0000","last_active":"2024-03-18T14:41:40.778484+0000","last_peered":"2024-03-18T14:41:40.778484+0000","last_clean":"2024-03-18T14:41:40.778484+0000","last_became_active":"2024-03-18T14:41:34.530415+0000","last_became_peered":"2024-03-18T14:41:34.530415+0000","last_unstale":"2024-03-18T14:41:40.778484+0000","last_undegraded":"2024-03-18T14:41:40.778484+0000","last_fullsized":"2024-03-18T14:41:40.778484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560204+0000","last_change":"2024-03-18T14:41:39.584982+0000","last_active":"2024-03-18T14:41:40.560204+0000","last_peered":"2024-03-18T14:41:40.560204+0000","last_clean":"2024-03-18T14:41:40.560204+0000","last_became_active":"2024-03-18T14:41:39.584630+0000","last_became_peered":"2024-03-18T14:41:39.584630+0000","last_unstale":"2024-03-18T14:41:40.560204+0000","last_undegraded":"2024-03-18T14:41:40.560204+0000","last_fullsized":"2024-03-18T14:41:40.560204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563830+0000","last_change":"2024-03-18T14:41:35.552305+0000","last_active":"2024-03-18T14:41:40.563830+0000","last_peered":"2024-03-18T14:41:40.563830+0000","last_clean":"2024-03-18T14:41:40.563830+0000","last_became_active":"2024-03-18T14:41:35.552031+0000","last_became_peered":"2024-03-18T14:41:35.552031+0000","last_unstale":"2024-03-18T14:41:40.563830+0000","last_undegraded":"2024-03-18T14:41:40.563830+0000","last_fullsized":"2024-03-18T14:41:40.563830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779939+0000","last_change":"2024-03-18T14:41:34.530930+0000","last_active":"2024-03-18T14:41:40.779939+0000","last_peered":"2024-03-18T14:41:40.779939+0000","last_clean":"2024-03-18T14:41:40.779939+0000","last_became_active":"2024-03-18T14:41:34.530645+0000","last_became_peered":"2024-03-18T14:41:34.530645+0000","last_unstale":"2024-03-18T14:41:40.779939+0000","last_undegraded":"2024-03-18T14:41:40.779939+0000","last_fullsized":"2024-03-18T14:41:40.779939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559457+0000","last_change":"2024-03-18T14:41:39.585926+0000","last_active":"2024-03-18T14:41:40.559457+0000","last_peered":"2024-03-18T14:41:40.559457+0000","last_clean":"2024-03-18T14:41:40.559457+0000","last_became_active":"2024-03-18T14:41:39.585465+0000","last_became_peered":"2024-03-18T14:41:39.585465+0000","last_unstale":"2024-03-18T14:41:40.559457+0000","last_undegraded":"2024-03-18T14:41:40.559457+0000","last_fullsized":"2024-03-18T14:41:40.559457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563386+0000","last_change":"2024-03-18T14:41:35.541836+0000","last_active":"2024-03-18T14:41:40.563386+0000","last_peered":"2024-03-18T14:41:40.563386+0000","last_clean":"2024-03-18T14:41:40.563386+0000","last_became_active":"2024-03-18T14:41:35.541532+0000","last_became_peered":"2024-03-18T14:41:35.541532+0000","last_unstale":"2024-03-18T14:41:40.563386+0000","last_undegraded":"2024-03-18T14:41:40.563386+0000","last_fullsized":"2024-03-18T14:41:40.563386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778143+0000","last_change":"2024-03-18T14:41:34.529169+0000","last_active":"2024-03-18T14:41:40.778143+0000","last_peered":"2024-03-18T14:41:40.778143+0000","last_clean":"2024-03-18T14:41:40.778143+0000","last_became_active":"2024-03-18T14:41:34.526870+0000","last_became_peered":"2024-03-18T14:41:34.526870+0000","last_unstale":"2024-03-18T14:41:40.778143+0000","last_undegraded":"2024-03-18T14:41:40.778143+0000","last_fullsized":"2024-03-18T14:41:40.778143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779714+0000","last_change":"2024-03-18T14:41:39.603928+0000","last_active":"2024-03-18T14:41:40.779714+0000","last_peered":"2024-03-18T14:41:40.779714+0000","last_clean":"2024-03-18T14:41:40.779714+0000","last_became_active":"2024-03-18T14:41:39.603236+0000","last_became_peered":"2024-03-18T14:41:39.603236+0000","last_unstale":"2024-03-18T14:41:40.779714+0000","last_undegraded":"2024-03-18T14:41:40.779714+0000","last_fullsized":"2024-03-18T14:41:40.779714+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559885+0000","last_change":"2024-03-18T14:41:35.538978+0000","last_active":"2024-03-18T14:41:40.559885+0000","last_peered":"2024-03-18T14:41:40.559885+0000","last_clean":"2024-03-18T14:41:40.559885+0000","last_became_active":"2024-03-18T14:41:35.538570+0000","last_became_peered":"2024-03-18T14:41:35.538570+0000","last_unstale":"2024-03-18T14:41:40.559885+0000","last_undegraded":"2024-03-18T14:41:40.559885+0000","last_fullsized":"2024-03-18T14:41:40.559885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:38.574781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777921+0000","last_change":"2024-03-18T14:41:34.537652+0000","last_active":"2024-03-18T14:41:40.777921+0000","last_peered":"2024-03-18T14:41:40.777921+0000","last_clean":"2024-03-18T14:41:40.777921+0000","last_became_active":"2024-03-18T14:41:34.537411+0000","last_became_peered":"2024-03-18T14:41:34.537411+0000","last_unstale":"2024-03-18T14:41:40.777921+0000","last_undegraded":"2024-03-18T14:41:40.777921+0000","last_fullsized":"2024-03-18T14:41:40.777921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560190+0000","last_change":"2024-03-18T14:41:39.580821+0000","last_active":"2024-03-18T14:41:40.560190+0000","last_peered":"2024-03-18T14:41:40.560190+0000","last_clean":"2024-03-18T14:41:40.560190+0000","last_became_active":"2024-03-18T14:41:39.580317+0000","last_became_peered":"2024-03-18T14:41:39.580317+0000","last_unstale":"2024-03-18T14:41:40.560190+0000","last_undegraded":"2024-03-18T14:41:40.560190+0000","last_fullsized":"2024-03-18T14:41:40.560190+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778799+0000","last_change":"2024-03-18T14:41:35.552139+0000","last_active":"2024-03-18T14:41:40.778799+0000","last_peered":"2024-03-18T14:41:40.778799+0000","last_clean":"2024-03-18T14:41:40.778799+0000","last_became_active":"2024-03-18T14:41:35.551809+0000","last_became_peered":"2024-03-18T14:41:35.551809+0000","last_unstale":"2024-03-18T14:41:40.778799+0000","last_undegraded":"2024-03-18T14:41:40.778799+0000","last_fullsized":"2024-03-18T14:41:40.778799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779124+0000","last_change":"2024-03-18T14:41:34.534282+0000","last_active":"2024-03-18T14:41:40.779124+0000","last_peered":"2024-03-18T14:41:40.779124+0000","last_clean":"2024-03-18T14:41:40.779124+0000","last_became_active":"2024-03-18T14:41:34.533937+0000","last_became_peered":"2024-03-18T14:41:34.533937+0000","last_unstale":"2024-03-18T14:41:40.779124+0000","last_undegraded":"2024-03-18T14:41:40.779124+0000","last_fullsized":"2024-03-18T14:41:40.779124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559601+0000","last_change":"2024-03-18T14:41:39.602565+0000","last_active":"2024-03-18T14:41:40.559601+0000","last_peered":"2024-03-18T14:41:40.559601+0000","last_clean":"2024-03-18T14:41:40.559601+0000","last_became_active":"2024-03-18T14:41:39.602164+0000","last_became_peered":"2024-03-18T14:41:39.602164+0000","last_unstale":"2024-03-18T14:41:40.559601+0000","last_undegraded":"2024-03-18T14:41:40.559601+0000","last_fullsized":"2024-03-18T14:41:40.559601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564632+0000","last_change":"2024-03-18T14:41:35.547600+0000","last_active":"2024-03-18T14:41:40.564632+0000","last_peered":"2024-03-18T14:41:40.564632+0000","last_clean":"2024-03-18T14:41:40.564632+0000","last_became_active":"2024-03-18T14:41:35.547324+0000","last_became_peered":"2024-03-18T14:41:35.547324+0000","last_unstale":"2024-03-18T14:41:40.564632+0000","last_undegraded":"2024-03-18T14:41:40.564632+0000","last_fullsized":"2024-03-18T14:41:40.564632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779055+0000","last_change":"2024-03-18T14:41:34.537441+0000","last_active":"2024-03-18T14:41:40.779055+0000","last_peered":"2024-03-18T14:41:40.779055+0000","last_clean":"2024-03-18T14:41:40.779055+0000","last_became_active":"2024-03-18T14:41:34.536599+0000","last_became_peered":"2024-03-18T14:41:34.536599+0000","last_unstale":"2024-03-18T14:41:40.779055+0000","last_undegraded":"2024-03-18T14:41:40.779055+0000","last_fullsized":"2024-03-18T14:41:40.779055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563656+0000","last_change":"2024-03-18T14:41:39.604429+0000","last_active":"2024-03-18T14:41:40.563656+0000","last_peered":"2024-03-18T14:41:40.563656+0000","last_clean":"2024-03-18T14:41:40.563656+0000","last_became_active":"2024-03-18T14:41:39.604046+0000","last_became_peered":"2024-03-18T14:41:39.604046+0000","last_unstale":"2024-03-18T14:41:40.563656+0000","last_undegraded":"2024-03-18T14:41:40.563656+0000","last_fullsized":"2024-03-18T14:41:40.563656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:28:21.835327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778649+0000","last_change":"2024-03-18T14:41:35.539912+0000","last_active":"2024-03-18T14:41:40.778649+0000","last_peered":"2024-03-18T14:41:40.778649+0000","last_clean":"2024-03-18T14:41:40.778649+0000","last_became_active":"2024-03-18T14:41:35.538765+0000","last_became_peered":"2024-03-18T14:41:35.538765+0000","last_unstale":"2024-03-18T14:41:40.778649+0000","last_undegraded":"2024-03-18T14:41:40.778649+0000","last_fullsized":"2024-03-18T14:41:40.778649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779021+0000","last_change":"2024-03-18T14:41:34.537358+0000","last_active":"2024-03-18T14:41:40.779021+0000","last_peered":"2024-03-18T14:41:40.779021+0000","last_clean":"2024-03-18T14:41:40.779021+0000","last_became_active":"2024-03-18T14:41:34.537020+0000","last_became_peered":"2024-03-18T14:41:34.537020+0000","last_unstale":"2024-03-18T14:41:40.779021+0000","last_undegraded":"2024-03-18T14:41:40.779021+0000","last_fullsized":"2024-03-18T14:41:40.779021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:24.398074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778734+0000","last_change":"2024-03-18T14:41:39.601563+0000","last_active":"2024-03-18T14:41:40.778734+0000","last_peered":"2024-03-18T14:41:40.778734+0000","last_clean":"2024-03-18T14:41:40.778734+0000","last_became_active":"2024-03-18T14:41:39.600878+0000","last_became_peered":"2024-03-18T14:41:39.600878+0000","last_unstale":"2024-03-18T14:41:40.778734+0000","last_undegraded":"2024-03-18T14:41:40.778734+0000","last_fullsized":"2024-03-18T14:41:40.778734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:41.241691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560348+0000","last_change":"2024-03-18T14:41:35.551415+0000","last_active":"2024-03-18T14:41:40.560348+0000","last_peered":"2024-03-18T14:41:40.560348+0000","last_clean":"2024-03-18T14:41:40.560348+0000","last_became_active":"2024-03-18T14:41:35.550751+0000","last_became_peered":"2024-03-18T14:41:35.550751+0000","last_unstale":"2024-03-18T14:41:40.560348+0000","last_undegraded":"2024-03-18T14:41:40.560348+0000","last_fullsized":"2024-03-18T14:41:40.560348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778749+0000","last_change":"2024-03-18T14:41:34.534249+0000","last_active":"2024-03-18T14:41:40.778749+0000","last_peered":"2024-03-18T14:41:40.778749+0000","last_clean":"2024-03-18T14:41:40.778749+0000","last_became_active":"2024-03-18T14:41:34.533913+0000","last_became_peered":"2024-03-18T14:41:34.533913+0000","last_unstale":"2024-03-18T14:41:40.778749+0000","last_undegraded":"2024-03-18T14:41:40.778749+0000","last_fullsized":"2024-03-18T14:41:40.778749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:39.582572+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:39.581652+0000","last_became_peered":"2024-03-18T14:41:39.581652+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:48.022171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560327+0000","last_change":"2024-03-18T14:41:35.561297+0000","last_active":"2024-03-18T14:41:40.560327+0000","last_peered":"2024-03-18T14:41:40.560327+0000","last_clean":"2024-03-18T14:41:40.560327+0000","last_became_active":"2024-03-18T14:41:35.558687+0000","last_became_peered":"2024-03-18T14:41:35.558687+0000","last_unstale":"2024-03-18T14:41:40.560327+0000","last_undegraded":"2024-03-18T14:41:40.560327+0000","last_fullsized":"2024-03-18T14:41:40.560327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:39:25.399581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559780+0000","last_change":"2024-03-18T14:41:34.525754+0000","last_active":"2024-03-18T14:41:40.559780+0000","last_peered":"2024-03-18T14:41:40.559780+0000","last_clean":"2024-03-18T14:41:40.559780+0000","last_became_active":"2024-03-18T14:41:34.525453+0000","last_became_peered":"2024-03-18T14:41:34.525453+0000","last_unstale":"2024-03-18T14:41:40.559780+0000","last_undegraded":"2024-03-18T14:41:40.559780+0000","last_fullsized":"2024-03-18T14:41:40.559780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778594+0000","last_change":"2024-03-18T14:41:39.584128+0000","last_active":"2024-03-18T14:41:40.778594+0000","last_peered":"2024-03-18T14:41:40.778594+0000","last_clean":"2024-03-18T14:41:40.778594+0000","last_became_active":"2024-03-18T14:41:39.582472+0000","last_became_peered":"2024-03-18T14:41:39.582472+0000","last_unstale":"2024-03-18T14:41:40.778594+0000","last_undegraded":"2024-03-18T14:41:40.778594+0000","last_fullsized":"2024-03-18T14:41:40.778594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778049+0000","last_change":"2024-03-18T14:41:35.540356+0000","last_active":"2024-03-18T14:41:40.778049+0000","last_peered":"2024-03-18T14:41:40.778049+0000","last_clean":"2024-03-18T14:41:40.778049+0000","last_became_active":"2024-03-18T14:41:35.540060+0000","last_became_peered":"2024-03-18T14:41:35.540060+0000","last_unstale":"2024-03-18T14:41:40.778049+0000","last_undegraded":"2024-03-18T14:41:40.778049+0000","last_fullsized":"2024-03-18T14:41:40.778049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778106+0000","last_change":"2024-03-18T14:41:34.529076+0000","last_active":"2024-03-18T14:41:40.778106+0000","last_peered":"2024-03-18T14:41:40.778106+0000","last_clean":"2024-03-18T14:41:40.778106+0000","last_became_active":"2024-03-18T14:41:34.528794+0000","last_became_peered":"2024-03-18T14:41:34.528794+0000","last_unstale":"2024-03-18T14:41:40.778106+0000","last_undegraded":"2024-03-18T14:41:40.778106+0000","last_fullsized":"2024-03-18T14:41:40.778106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779855+0000","last_change":"2024-03-18T14:41:39.581385+0000","last_active":"2024-03-18T14:41:40.779855+0000","last_peered":"2024-03-18T14:41:40.779855+0000","last_clean":"2024-03-18T14:41:40.779855+0000","last_became_active":"2024-03-18T14:41:39.580972+0000","last_became_peered":"2024-03-18T14:41:39.580972+0000","last_unstale":"2024-03-18T14:41:40.779855+0000","last_undegraded":"2024-03-18T14:41:40.779855+0000","last_fullsized":"2024-03-18T14:41:40.779855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778334+0000","last_change":"2024-03-18T14:41:35.552692+0000","last_active":"2024-03-18T14:41:40.778334+0000","last_peered":"2024-03-18T14:41:40.778334+0000","last_clean":"2024-03-18T14:41:40.778334+0000","last_became_active":"2024-03-18T14:41:35.552340+0000","last_became_peered":"2024-03-18T14:41:35.552340+0000","last_unstale":"2024-03-18T14:41:40.778334+0000","last_undegraded":"2024-03-18T14:41:40.778334+0000","last_fullsized":"2024-03-18T14:41:40.778334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778015+0000","last_change":"2024-03-18T14:41:34.531973+0000","last_active":"2024-03-18T14:41:40.778015+0000","last_peered":"2024-03-18T14:41:40.778015+0000","last_clean":"2024-03-18T14:41:40.778015+0000","last_became_active":"2024-03-18T14:41:34.531796+0000","last_became_peered":"2024-03-18T14:41:34.531796+0000","last_unstale":"2024-03-18T14:41:40.778015+0000","last_undegraded":"2024-03-18T14:41:40.778015+0000","last_fullsized":"2024-03-18T14:41:40.778015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:37.573273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778621+0000","last_change":"2024-03-18T14:41:39.601287+0000","last_active":"2024-03-18T14:41:40.778621+0000","last_peered":"2024-03-18T14:41:40.778621+0000","last_clean":"2024-03-18T14:41:40.778621+0000","last_became_active":"2024-03-18T14:41:39.600855+0000","last_became_peered":"2024-03-18T14:41:39.600855+0000","last_unstale":"2024-03-18T14:41:40.778621+0000","last_undegraded":"2024-03-18T14:41:40.778621+0000","last_fullsized":"2024-03-18T14:41:40.778621+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:26.606447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563301+0000","last_change":"2024-03-18T14:41:35.547848+0000","last_active":"2024-03-18T14:41:40.563301+0000","last_peered":"2024-03-18T14:41:40.563301+0000","last_clean":"2024-03-18T14:41:40.563301+0000","last_became_active":"2024-03-18T14:41:35.547437+0000","last_became_peered":"2024-03-18T14:41:35.547437+0000","last_unstale":"2024-03-18T14:41:40.563301+0000","last_undegraded":"2024-03-18T14:41:40.563301+0000","last_fullsized":"2024-03-18T14:41:40.563301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559408+0000","last_change":"2024-03-18T14:41:34.528039+0000","last_active":"2024-03-18T14:41:40.559408+0000","last_peered":"2024-03-18T14:41:40.559408+0000","last_clean":"2024-03-18T14:41:40.559408+0000","last_became_active":"2024-03-18T14:41:34.527756+0000","last_became_peered":"2024-03-18T14:41:34.527756+0000","last_unstale":"2024-03-18T14:41:40.559408+0000","last_undegraded":"2024-03-18T14:41:40.559408+0000","last_fullsized":"2024-03-18T14:41:40.559408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778101+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:41:40.778101+0000","last_peered":"2024-03-18T14:41:40.778101+0000","last_clean":"2024-03-18T14:41:40.778101+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:41:40.778101+0000","last_undegraded":"2024-03-18T14:41:40.778101+0000","last_fullsized":"2024-03-18T14:41:40.778101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778699+0000","last_change":"2024-03-18T14:41:39.602108+0000","last_active":"2024-03-18T14:41:40.778699+0000","last_peered":"2024-03-18T14:41:40.778699+0000","last_clean":"2024-03-18T14:41:40.778699+0000","last_became_active":"2024-03-18T14:41:39.601732+0000","last_became_peered":"2024-03-18T14:41:39.601732+0000","last_unstale":"2024-03-18T14:41:40.778699+0000","last_undegraded":"2024-03-18T14:41:40.778699+0000","last_fullsized":"2024-03-18T14:41:40.778699+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:11.189905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778841+0000","last_change":"2024-03-18T14:41:35.544776+0000","last_active":"2024-03-18T14:41:40.778841+0000","last_peered":"2024-03-18T14:41:40.778841+0000","last_clean":"2024-03-18T14:41:40.778841+0000","last_became_active":"2024-03-18T14:41:35.544498+0000","last_became_peered":"2024-03-18T14:41:35.544498+0000","last_unstale":"2024-03-18T14:41:40.778841+0000","last_undegraded":"2024-03-18T14:41:40.778841+0000","last_fullsized":"2024-03-18T14:41:40.778841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780098+0000","last_change":"2024-03-18T14:41:34.531645+0000","last_active":"2024-03-18T14:41:40.780098+0000","last_peered":"2024-03-18T14:41:40.780098+0000","last_clean":"2024-03-18T14:41:40.780098+0000","last_became_active":"2024-03-18T14:41:34.530361+0000","last_became_peered":"2024-03-18T14:41:34.530361+0000","last_unstale":"2024-03-18T14:41:40.780098+0000","last_undegraded":"2024-03-18T14:41:40.780098+0000","last_fullsized":"2024-03-18T14:41:40.780098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778063+0000","last_change":"2024-03-18T14:41:39.601190+0000","last_active":"2024-03-18T14:41:40.778063+0000","last_peered":"2024-03-18T14:41:40.778063+0000","last_clean":"2024-03-18T14:41:40.778063+0000","last_became_active":"2024-03-18T14:41:39.600809+0000","last_became_peered":"2024-03-18T14:41:39.600809+0000","last_unstale":"2024-03-18T14:41:40.778063+0000","last_undegraded":"2024-03-18T14:41:40.778063+0000","last_fullsized":"2024-03-18T14:41:40.778063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:12.679477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559630+0000","last_change":"2024-03-18T14:41:35.549443+0000","last_active":"2024-03-18T14:41:40.559630+0000","last_peered":"2024-03-18T14:41:40.559630+0000","last_clean":"2024-03-18T14:41:40.559630+0000","last_became_active":"2024-03-18T14:41:35.546376+0000","last_became_peered":"2024-03-18T14:41:35.546376+0000","last_unstale":"2024-03-18T14:41:40.559630+0000","last_undegraded":"2024-03-18T14:41:40.559630+0000","last_fullsized":"2024-03-18T14:41:40.559630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779349+0000","last_change":"2024-03-18T14:41:34.528408+0000","last_active":"2024-03-18T14:41:40.779349+0000","last_peered":"2024-03-18T14:41:40.779349+0000","last_clean":"2024-03-18T14:41:40.779349+0000","last_became_active":"2024-03-18T14:41:34.527981+0000","last_became_peered":"2024-03-18T14:41:34.527981+0000","last_unstale":"2024-03-18T14:41:40.779349+0000","last_undegraded":"2024-03-18T14:41:40.779349+0000","last_fullsized":"2024-03-18T14:41:40.779349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778976+0000","last_change":"2024-03-18T14:41:39.586214+0000","last_active":"2024-03-18T14:41:40.778976+0000","last_peered":"2024-03-18T14:41:40.778976+0000","last_clean":"2024-03-18T14:41:40.778976+0000","last_became_active":"2024-03-18T14:41:39.585773+0000","last_became_peered":"2024-03-18T14:41:39.585773+0000","last_unstale":"2024-03-18T14:41:40.778976+0000","last_undegraded":"2024-03-18T14:41:40.778976+0000","last_fullsized":"2024-03-18T14:41:40.778976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:52:45.131607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778850+0000","last_change":"2024-03-18T14:41:35.553371+0000","last_active":"2024-03-18T14:41:40.778850+0000","last_peered":"2024-03-18T14:41:40.778850+0000","last_clean":"2024-03-18T14:41:40.778850+0000","last_became_active":"2024-03-18T14:41:35.552906+0000","last_became_peered":"2024-03-18T14:41:35.552906+0000","last_unstale":"2024-03-18T14:41:40.778850+0000","last_undegraded":"2024-03-18T14:41:40.778850+0000","last_fullsized":"2024-03-18T14:41:40.778850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:22.569526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778853+0000","last_change":"2024-03-18T14:41:34.532843+0000","last_active":"2024-03-18T14:41:40.778853+0000","last_peered":"2024-03-18T14:41:40.778853+0000","last_clean":"2024-03-18T14:41:40.778853+0000","last_became_active":"2024-03-18T14:41:34.532183+0000","last_became_peered":"2024-03-18T14:41:34.532183+0000","last_unstale":"2024-03-18T14:41:40.778853+0000","last_undegraded":"2024-03-18T14:41:40.778853+0000","last_fullsized":"2024-03-18T14:41:40.778853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559016+0000","last_change":"2024-03-18T14:41:39.603278+0000","last_active":"2024-03-18T14:41:40.559016+0000","last_peered":"2024-03-18T14:41:40.559016+0000","last_clean":"2024-03-18T14:41:40.559016+0000","last_became_active":"2024-03-18T14:41:39.602927+0000","last_became_peered":"2024-03-18T14:41:39.602927+0000","last_unstale":"2024-03-18T14:41:40.559016+0000","last_undegraded":"2024-03-18T14:41:40.559016+0000","last_fullsized":"2024-03-18T14:41:40.559016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778747+0000","last_change":"2024-03-18T14:41:35.552924+0000","last_active":"2024-03-18T14:41:40.778747+0000","last_peered":"2024-03-18T14:41:40.778747+0000","last_clean":"2024-03-18T14:41:40.778747+0000","last_became_active":"2024-03-18T14:41:35.552514+0000","last_became_peered":"2024-03-18T14:41:35.552514+0000","last_unstale":"2024-03-18T14:41:40.778747+0000","last_undegraded":"2024-03-18T14:41:40.778747+0000","last_fullsized":"2024-03-18T14:41:40.778747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560464+0000","last_change":"2024-03-18T14:41:34.534565+0000","last_active":"2024-03-18T14:41:40.560464+0000","last_peered":"2024-03-18T14:41:40.560464+0000","last_clean":"2024-03-18T14:41:40.560464+0000","last_became_active":"2024-03-18T14:41:34.534302+0000","last_became_peered":"2024-03-18T14:41:34.534302+0000","last_unstale":"2024-03-18T14:41:40.560464+0000","last_undegraded":"2024-03-18T14:41:40.560464+0000","last_fullsized":"2024-03-18T14:41:40.560464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559373+0000","last_change":"2024-03-18T14:41:39.580698+0000","last_active":"2024-03-18T14:41:40.559373+0000","last_peered":"2024-03-18T14:41:40.559373+0000","last_clean":"2024-03-18T14:41:40.559373+0000","last_became_active":"2024-03-18T14:41:39.580161+0000","last_became_peered":"2024-03-18T14:41:39.580161+0000","last_unstale":"2024-03-18T14:41:40.559373+0000","last_undegraded":"2024-03-18T14:41:40.559373+0000","last_fullsized":"2024-03-18T14:41:40.559373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:03.265239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778352+0000","last_change":"2024-03-18T14:41:35.547876+0000","last_active":"2024-03-18T14:41:40.778352+0000","last_peered":"2024-03-18T14:41:40.778352+0000","last_clean":"2024-03-18T14:41:40.778352+0000","last_became_active":"2024-03-18T14:41:35.547615+0000","last_became_peered":"2024-03-18T14:41:35.547615+0000","last_unstale":"2024-03-18T14:41:40.778352+0000","last_undegraded":"2024-03-18T14:41:40.778352+0000","last_fullsized":"2024-03-18T14:41:40.778352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777856+0000","last_change":"2024-03-18T14:41:34.526247+0000","last_active":"2024-03-18T14:41:40.777856+0000","last_peered":"2024-03-18T14:41:40.777856+0000","last_clean":"2024-03-18T14:41:40.777856+0000","last_became_active":"2024-03-18T14:41:34.525952+0000","last_became_peered":"2024-03-18T14:41:34.525952+0000","last_unstale":"2024-03-18T14:41:40.777856+0000","last_undegraded":"2024-03-18T14:41:40.777856+0000","last_fullsized":"2024-03-18T14:41:40.777856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:39.580945+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:39.580421+0000","last_became_peered":"2024-03-18T14:41:39.580421+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778016+0000","last_change":"2024-03-18T14:41:35.554425+0000","last_active":"2024-03-18T14:41:40.778016+0000","last_peered":"2024-03-18T14:41:40.778016+0000","last_clean":"2024-03-18T14:41:40.778016+0000","last_became_active":"2024-03-18T14:41:35.554164+0000","last_became_peered":"2024-03-18T14:41:35.554164+0000","last_unstale":"2024-03-18T14:41:40.778016+0000","last_undegraded":"2024-03-18T14:41:40.778016+0000","last_fullsized":"2024-03-18T14:41:40.778016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778159+0000","last_change":"2024-03-18T14:41:34.528726+0000","last_active":"2024-03-18T14:41:40.778159+0000","last_peered":"2024-03-18T14:41:40.778159+0000","last_clean":"2024-03-18T14:41:40.778159+0000","last_became_active":"2024-03-18T14:41:34.528424+0000","last_became_peered":"2024-03-18T14:41:34.528424+0000","last_unstale":"2024-03-18T14:41:40.778159+0000","last_undegraded":"2024-03-18T14:41:40.778159+0000","last_fullsized":"2024-03-18T14:41:40.778159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:24.672529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564629+0000","last_change":"2024-03-18T14:41:39.602221+0000","last_active":"2024-03-18T14:41:40.564629+0000","last_peered":"2024-03-18T14:41:40.564629+0000","last_clean":"2024-03-18T14:41:40.564629+0000","last_became_active":"2024-03-18T14:41:39.601652+0000","last_became_peered":"2024-03-18T14:41:39.601652+0000","last_unstale":"2024-03-18T14:41:40.564629+0000","last_undegraded":"2024-03-18T14:41:40.564629+0000","last_fullsized":"2024-03-18T14:41:40.564629+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:00.363235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778400+0000","last_change":"2024-03-18T14:41:35.548474+0000","last_active":"2024-03-18T14:41:40.778400+0000","last_peered":"2024-03-18T14:41:40.778400+0000","last_clean":"2024-03-18T14:41:40.778400+0000","last_became_active":"2024-03-18T14:41:35.548193+0000","last_became_peered":"2024-03-18T14:41:35.548193+0000","last_unstale":"2024-03-18T14:41:40.778400+0000","last_undegraded":"2024-03-18T14:41:40.778400+0000","last_fullsized":"2024-03-18T14:41:40.778400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559701+0000","last_change":"2024-03-18T14:41:34.529770+0000","last_active":"2024-03-18T14:41:40.559701+0000","last_peered":"2024-03-18T14:41:40.559701+0000","last_clean":"2024-03-18T14:41:40.559701+0000","last_became_active":"2024-03-18T14:41:34.529499+0000","last_became_peered":"2024-03-18T14:41:34.529499+0000","last_unstale":"2024-03-18T14:41:40.559701+0000","last_undegraded":"2024-03-18T14:41:40.559701+0000","last_fullsized":"2024-03-18T14:41:40.559701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564084+0000","last_change":"2024-03-18T14:41:39.601622+0000","last_active":"2024-03-18T14:41:40.564084+0000","last_peered":"2024-03-18T14:41:40.564084+0000","last_clean":"2024-03-18T14:41:40.564084+0000","last_became_active":"2024-03-18T14:41:39.601209+0000","last_became_peered":"2024-03-18T14:41:39.601209+0000","last_unstale":"2024-03-18T14:41:40.564084+0000","last_undegraded":"2024-03-18T14:41:40.564084+0000","last_fullsized":"2024-03-18T14:41:40.564084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:11.443644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778881+0000","last_change":"2024-03-18T14:41:35.547642+0000","last_active":"2024-03-18T14:41:40.778881+0000","last_peered":"2024-03-18T14:41:40.778881+0000","last_clean":"2024-03-18T14:41:40.778881+0000","last_became_active":"2024-03-18T14:41:35.547354+0000","last_became_peered":"2024-03-18T14:41:35.547354+0000","last_unstale":"2024-03-18T14:41:40.778881+0000","last_undegraded":"2024-03-18T14:41:40.778881+0000","last_fullsized":"2024-03-18T14:41:40.778881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779200+0000","last_change":"2024-03-18T14:41:34.533998+0000","last_active":"2024-03-18T14:41:40.779200+0000","last_peered":"2024-03-18T14:41:40.779200+0000","last_clean":"2024-03-18T14:41:40.779200+0000","last_became_active":"2024-03-18T14:41:34.533597+0000","last_became_peered":"2024-03-18T14:41:34.533597+0000","last_unstale":"2024-03-18T14:41:40.779200+0000","last_undegraded":"2024-03-18T14:41:40.779200+0000","last_fullsized":"2024-03-18T14:41:40.779200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560211+0000","last_change":"2024-03-18T14:41:39.601126+0000","last_active":"2024-03-18T14:41:40.560211+0000","last_peered":"2024-03-18T14:41:40.560211+0000","last_clean":"2024-03-18T14:41:40.560211+0000","last_became_active":"2024-03-18T14:41:39.600751+0000","last_became_peered":"2024-03-18T14:41:39.600751+0000","last_unstale":"2024-03-18T14:41:40.560211+0000","last_undegraded":"2024-03-18T14:41:40.560211+0000","last_fullsized":"2024-03-18T14:41:40.560211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778816+0000","last_change":"2024-03-18T14:41:35.550195+0000","last_active":"2024-03-18T14:41:40.778816+0000","last_peered":"2024-03-18T14:41:40.778816+0000","last_clean":"2024-03-18T14:41:40.778816+0000","last_became_active":"2024-03-18T14:41:35.549894+0000","last_became_peered":"2024-03-18T14:41:35.549894+0000","last_unstale":"2024-03-18T14:41:40.778816+0000","last_undegraded":"2024-03-18T14:41:40.778816+0000","last_fullsized":"2024-03-18T14:41:40.778816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:45.045404+0000","last_change":"2024-03-18T14:41:34.536841+0000","last_active":"2024-03-18T14:41:45.045404+0000","last_peered":"2024-03-18T14:41:45.045404+0000","last_clean":"2024-03-18T14:41:45.045404+0000","last_became_active":"2024-03-18T14:41:34.536536+0000","last_became_peered":"2024-03-18T14:41:34.536536+0000","last_unstale":"2024-03-18T14:41:45.045404+0000","last_undegraded":"2024-03-18T14:41:45.045404+0000","last_fullsized":"2024-03-18T14:41:45.045404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:13:40.621847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779808+0000","last_change":"2024-03-18T14:41:39.582692+0000","last_active":"2024-03-18T14:41:40.779808+0000","last_peered":"2024-03-18T14:41:40.779808+0000","last_clean":"2024-03-18T14:41:40.779808+0000","last_became_active":"2024-03-18T14:41:39.582285+0000","last_became_peered":"2024-03-18T14:41:39.582285+0000","last_unstale":"2024-03-18T14:41:40.779808+0000","last_undegraded":"2024-03-18T14:41:40.779808+0000","last_fullsized":"2024-03-18T14:41:40.779808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:04.011866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559959+0000","last_change":"2024-03-18T14:41:35.547692+0000","last_active":"2024-03-18T14:41:40.559959+0000","last_peered":"2024-03-18T14:41:40.559959+0000","last_clean":"2024-03-18T14:41:40.559959+0000","last_became_active":"2024-03-18T14:41:35.547381+0000","last_became_peered":"2024-03-18T14:41:35.547381+0000","last_unstale":"2024-03-18T14:41:40.559959+0000","last_undegraded":"2024-03-18T14:41:40.559959+0000","last_fullsized":"2024-03-18T14:41:40.559959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:13.037877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779251+0000","last_change":"2024-03-18T14:41:34.529622+0000","last_active":"2024-03-18T14:41:40.779251+0000","last_peered":"2024-03-18T14:41:40.779251+0000","last_clean":"2024-03-18T14:41:40.779251+0000","last_became_active":"2024-03-18T14:41:34.528752+0000","last_became_peered":"2024-03-18T14:41:34.528752+0000","last_unstale":"2024-03-18T14:41:40.779251+0000","last_undegraded":"2024-03-18T14:41:40.779251+0000","last_fullsized":"2024-03-18T14:41:40.779251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:36.508308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560121+0000","last_change":"2024-03-18T14:41:39.585230+0000","last_active":"2024-03-18T14:41:40.560121+0000","last_peered":"2024-03-18T14:41:40.560121+0000","last_clean":"2024-03-18T14:41:40.560121+0000","last_became_active":"2024-03-18T14:41:39.584896+0000","last_became_peered":"2024-03-18T14:41:39.584896+0000","last_unstale":"2024-03-18T14:41:40.560121+0000","last_undegraded":"2024-03-18T14:41:40.560121+0000","last_fullsized":"2024-03-18T14:41:40.560121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779508+0000","last_change":"2024-03-18T14:41:35.552508+0000","last_active":"2024-03-18T14:41:40.779508+0000","last_peered":"2024-03-18T14:41:40.779508+0000","last_clean":"2024-03-18T14:41:40.779508+0000","last_became_active":"2024-03-18T14:41:35.552221+0000","last_became_peered":"2024-03-18T14:41:35.552221+0000","last_unstale":"2024-03-18T14:41:40.779508+0000","last_undegraded":"2024-03-18T14:41:40.779508+0000","last_fullsized":"2024-03-18T14:41:40.779508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560164+0000","last_change":"2024-03-18T14:41:34.534226+0000","last_active":"2024-03-18T14:41:40.560164+0000","last_peered":"2024-03-18T14:41:40.560164+0000","last_clean":"2024-03-18T14:41:40.560164+0000","last_became_active":"2024-03-18T14:41:34.533977+0000","last_became_peered":"2024-03-18T14:41:34.533977+0000","last_unstale":"2024-03-18T14:41:40.560164+0000","last_undegraded":"2024-03-18T14:41:40.560164+0000","last_fullsized":"2024-03-18T14:41:40.560164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563415+0000","last_change":"2024-03-18T14:41:39.583657+0000","last_active":"2024-03-18T14:41:40.563415+0000","last_peered":"2024-03-18T14:41:40.563415+0000","last_clean":"2024-03-18T14:41:40.563415+0000","last_became_active":"2024-03-18T14:41:39.583280+0000","last_became_peered":"2024-03-18T14:41:39.583280+0000","last_unstale":"2024-03-18T14:41:40.563415+0000","last_undegraded":"2024-03-18T14:41:40.563415+0000","last_fullsized":"2024-03-18T14:41:40.563415+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:48.355078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560126+0000","last_change":"2024-03-18T14:41:35.546368+0000","last_active":"2024-03-18T14:41:40.560126+0000","last_peered":"2024-03-18T14:41:40.560126+0000","last_clean":"2024-03-18T14:41:40.560126+0000","last_became_active":"2024-03-18T14:41:35.546101+0000","last_became_peered":"2024-03-18T14:41:35.546101+0000","last_unstale":"2024-03-18T14:41:40.560126+0000","last_undegraded":"2024-03-18T14:41:40.560126+0000","last_fullsized":"2024-03-18T14:41:40.560126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:25.504913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779013+0000","last_change":"2024-03-18T14:41:34.535621+0000","last_active":"2024-03-18T14:41:40.779013+0000","last_peered":"2024-03-18T14:41:40.779013+0000","last_clean":"2024-03-18T14:41:40.779013+0000","last_became_active":"2024-03-18T14:41:34.535351+0000","last_became_peered":"2024-03-18T14:41:34.535351+0000","last_unstale":"2024-03-18T14:41:40.779013+0000","last_undegraded":"2024-03-18T14:41:40.779013+0000","last_fullsized":"2024-03-18T14:41:40.779013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:33.941965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563431+0000","last_change":"2024-03-18T14:41:39.602374+0000","last_active":"2024-03-18T14:41:40.563431+0000","last_peered":"2024-03-18T14:41:40.563431+0000","last_clean":"2024-03-18T14:41:40.563431+0000","last_became_active":"2024-03-18T14:41:39.600913+0000","last_became_peered":"2024-03-18T14:41:39.600913+0000","last_unstale":"2024-03-18T14:41:40.563431+0000","last_undegraded":"2024-03-18T14:41:40.563431+0000","last_fullsized":"2024-03-18T14:41:40.563431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:24:45.565111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778290+0000","last_change":"2024-03-18T14:41:39.602659+0000","last_active":"2024-03-18T14:41:40.778290+0000","last_peered":"2024-03-18T14:41:40.778290+0000","last_clean":"2024-03-18T14:41:40.778290+0000","last_became_active":"2024-03-18T14:41:39.602286+0000","last_became_peered":"2024-03-18T14:41:39.602286+0000","last_unstale":"2024-03-18T14:41:40.778290+0000","last_undegraded":"2024-03-18T14:41:40.778290+0000","last_fullsized":"2024-03-18T14:41:40.778290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:29.541834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778038+0000","last_change":"2024-03-18T14:41:35.545020+0000","last_active":"2024-03-18T14:41:40.778038+0000","last_peered":"2024-03-18T14:41:40.778038+0000","last_clean":"2024-03-18T14:41:40.778038+0000","last_became_active":"2024-03-18T14:41:35.544697+0000","last_became_peered":"2024-03-18T14:41:35.544697+0000","last_unstale":"2024-03-18T14:41:40.778038+0000","last_undegraded":"2024-03-18T14:41:40.778038+0000","last_fullsized":"2024-03-18T14:41:40.778038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559154+0000","last_change":"2024-03-18T14:41:34.527624+0000","last_active":"2024-03-18T14:41:40.559154+0000","last_peered":"2024-03-18T14:41:40.559154+0000","last_clean":"2024-03-18T14:41:40.559154+0000","last_became_active":"2024-03-18T14:41:34.527291+0000","last_became_peered":"2024-03-18T14:41:34.527291+0000","last_unstale":"2024-03-18T14:41:40.559154+0000","last_undegraded":"2024-03-18T14:41:40.559154+0000","last_fullsized":"2024-03-18T14:41:40.559154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:56.597027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779046+0000","last_change":"2024-03-18T14:41:39.575946+0000","last_active":"2024-03-18T14:41:40.779046+0000","last_peered":"2024-03-18T14:41:40.779046+0000","last_clean":"2024-03-18T14:41:40.779046+0000","last_became_active":"2024-03-18T14:41:39.575618+0000","last_became_peered":"2024-03-18T14:41:39.575618+0000","last_unstale":"2024-03-18T14:41:40.779046+0000","last_undegraded":"2024-03-18T14:41:40.779046+0000","last_fullsized":"2024-03-18T14:41:40.779046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:40.782715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563258+0000","last_change":"2024-03-18T14:41:35.548573+0000","last_active":"2024-03-18T14:41:40.563258+0000","last_peered":"2024-03-18T14:41:40.563258+0000","last_clean":"2024-03-18T14:41:40.563258+0000","last_became_active":"2024-03-18T14:41:35.548284+0000","last_became_peered":"2024-03-18T14:41:35.548284+0000","last_unstale":"2024-03-18T14:41:40.563258+0000","last_undegraded":"2024-03-18T14:41:40.563258+0000","last_fullsized":"2024-03-18T14:41:40.563258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563616+0000","last_change":"2024-03-18T14:41:34.536158+0000","last_active":"2024-03-18T14:41:40.563616+0000","last_peered":"2024-03-18T14:41:40.563616+0000","last_clean":"2024-03-18T14:41:40.563616+0000","last_became_active":"2024-03-18T14:41:34.535879+0000","last_became_peered":"2024-03-18T14:41:34.535879+0000","last_unstale":"2024-03-18T14:41:40.563616+0000","last_undegraded":"2024-03-18T14:41:40.563616+0000","last_fullsized":"2024-03-18T14:41:40.563616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:36:12.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778365+0000","last_change":"2024-03-18T14:41:34.528073+0000","last_active":"2024-03-18T14:41:40.778365+0000","last_peered":"2024-03-18T14:41:40.778365+0000","last_clean":"2024-03-18T14:41:40.778365+0000","last_became_active":"2024-03-18T14:41:34.527245+0000","last_became_peered":"2024-03-18T14:41:34.527245+0000","last_unstale":"2024-03-18T14:41:40.778365+0000","last_undegraded":"2024-03-18T14:41:40.778365+0000","last_fullsized":"2024-03-18T14:41:40.778365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:14.628222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779106+0000","last_change":"2024-03-18T14:41:35.560334+0000","last_active":"2024-03-18T14:41:40.779106+0000","last_peered":"2024-03-18T14:41:40.779106+0000","last_clean":"2024-03-18T14:41:40.779106+0000","last_became_active":"2024-03-18T14:41:35.559938+0000","last_became_peered":"2024-03-18T14:41:35.559938+0000","last_unstale":"2024-03-18T14:41:40.779106+0000","last_undegraded":"2024-03-18T14:41:40.779106+0000","last_fullsized":"2024-03-18T14:41:40.779106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.780010+0000","last_change":"2024-03-18T14:41:39.578410+0000","last_active":"2024-03-18T14:41:40.780010+0000","last_peered":"2024-03-18T14:41:40.780010+0000","last_clean":"2024-03-18T14:41:40.780010+0000","last_became_active":"2024-03-18T14:41:39.577919+0000","last_became_peered":"2024-03-18T14:41:39.577919+0000","last_unstale":"2024-03-18T14:41:40.780010+0000","last_undegraded":"2024-03-18T14:41:40.780010+0000","last_fullsized":"2024-03-18T14:41:40.780010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:59.569962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563940+0000","last_change":"2024-03-18T14:41:34.535233+0000","last_active":"2024-03-18T14:41:40.563940+0000","last_peered":"2024-03-18T14:41:40.563940+0000","last_clean":"2024-03-18T14:41:40.563940+0000","last_became_active":"2024-03-18T14:41:34.534879+0000","last_became_peered":"2024-03-18T14:41:34.534879+0000","last_unstale":"2024-03-18T14:41:40.563940+0000","last_undegraded":"2024-03-18T14:41:40.563940+0000","last_fullsized":"2024-03-18T14:41:40.563940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:33.508461+0000","last_clean_scrub_stamp":"2024-03-18T14:41:33.508461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:26:45.208318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559603+0000","last_change":"2024-03-18T14:41:35.535546+0000","last_active":"2024-03-18T14:41:40.559603+0000","last_peered":"2024-03-18T14:41:40.559603+0000","last_clean":"2024-03-18T14:41:40.559603+0000","last_became_active":"2024-03-18T14:41:35.535269+0000","last_became_peered":"2024-03-18T14:41:35.535269+0000","last_unstale":"2024-03-18T14:41:40.559603+0000","last_undegraded":"2024-03-18T14:41:40.559603+0000","last_fullsized":"2024-03-18T14:41:40.559603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564263+0000","last_change":"2024-03-18T14:41:39.583244+0000","last_active":"2024-03-18T14:41:40.564263+0000","last_peered":"2024-03-18T14:41:40.564263+0000","last_clean":"2024-03-18T14:41:40.564263+0000","last_became_active":"2024-03-18T14:41:39.582882+0000","last_became_peered":"2024-03-18T14:41:39.582882+0000","last_unstale":"2024-03-18T14:41:40.564263+0000","last_undegraded":"2024-03-18T14:41:40.564263+0000","last_fullsized":"2024-03-18T14:41:40.564263+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:38.546890+0000","last_clean_scrub_stamp":"2024-03-18T14:41:38.546890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:10:48.564938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777919+0000","last_change":"2024-03-18T14:41:35.554708+0000","last_active":"2024-03-18T14:41:40.777919+0000","last_peered":"2024-03-18T14:41:40.777919+0000","last_clean":"2024-03-18T14:41:40.777919+0000","last_became_active":"2024-03-18T14:41:35.554411+0000","last_became_peered":"2024-03-18T14:41:35.554411+0000","last_unstale":"2024-03-18T14:41:40.777919+0000","last_undegraded":"2024-03-18T14:41:40.777919+0000","last_fullsized":"2024-03-18T14:41:40.777919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.777933+0000","last_change":"2024-03-18T14:41:35.543401+0000","last_active":"2024-03-18T14:41:40.777933+0000","last_peered":"2024-03-18T14:41:40.777933+0000","last_clean":"2024-03-18T14:41:40.777933+0000","last_became_active":"2024-03-18T14:41:35.543034+0000","last_became_peered":"2024-03-18T14:41:35.543034+0000","last_unstale":"2024-03-18T14:41:40.777933+0000","last_undegraded":"2024-03-18T14:41:40.777933+0000","last_fullsized":"2024-03-18T14:41:40.777933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:57.598535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559616+0000","last_change":"2024-03-18T14:41:35.551162+0000","last_active":"2024-03-18T14:41:40.559616+0000","last_peered":"2024-03-18T14:41:40.559616+0000","last_clean":"2024-03-18T14:41:40.559616+0000","last_became_active":"2024-03-18T14:41:35.550839+0000","last_became_peered":"2024-03-18T14:41:35.550839+0000","last_unstale":"2024-03-18T14:41:40.559616+0000","last_undegraded":"2024-03-18T14:41:40.559616+0000","last_fullsized":"2024-03-18T14:41:40.559616+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:21:08.642556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779023+0000","last_change":"2024-03-18T14:41:35.554787+0000","last_active":"2024-03-18T14:41:40.779023+0000","last_peered":"2024-03-18T14:41:40.779023+0000","last_clean":"2024-03-18T14:41:40.779023+0000","last_became_active":"2024-03-18T14:41:35.554484+0000","last_became_peered":"2024-03-18T14:41:35.554484+0000","last_unstale":"2024-03-18T14:41:40.779023+0000","last_undegraded":"2024-03-18T14:41:40.779023+0000","last_fullsized":"2024-03-18T14:41:40.779023+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:14.506229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564088+0000","last_change":"2024-03-18T14:41:35.544958+0000","last_active":"2024-03-18T14:41:40.564088+0000","last_peered":"2024-03-18T14:41:40.564088+0000","last_clean":"2024-03-18T14:41:40.564088+0000","last_became_active":"2024-03-18T14:41:35.544600+0000","last_became_peered":"2024-03-18T14:41:35.544600+0000","last_unstale":"2024-03-18T14:41:40.564088+0000","last_undegraded":"2024-03-18T14:41:40.564088+0000","last_fullsized":"2024-03-18T14:41:40.564088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778046+0000","last_change":"2024-03-18T14:41:35.549309+0000","last_active":"2024-03-18T14:41:40.778046+0000","last_peered":"2024-03-18T14:41:40.778046+0000","last_clean":"2024-03-18T14:41:40.778046+0000","last_became_active":"2024-03-18T14:41:35.549039+0000","last_became_peered":"2024-03-18T14:41:35.549039+0000","last_unstale":"2024-03-18T14:41:40.778046+0000","last_undegraded":"2024-03-18T14:41:40.778046+0000","last_fullsized":"2024-03-18T14:41:40.778046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778908+0000","last_change":"2024-03-18T14:41:35.563387+0000","last_active":"2024-03-18T14:41:40.778908+0000","last_peered":"2024-03-18T14:41:40.778908+0000","last_clean":"2024-03-18T14:41:40.778908+0000","last_became_active":"2024-03-18T14:41:35.562621+0000","last_became_peered":"2024-03-18T14:41:35.562621+0000","last_unstale":"2024-03-18T14:41:40.778908+0000","last_undegraded":"2024-03-18T14:41:40.778908+0000","last_fullsized":"2024-03-18T14:41:40.778908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.564514+0000","last_change":"2024-03-18T14:41:35.549988+0000","last_active":"2024-03-18T14:41:40.564514+0000","last_peered":"2024-03-18T14:41:40.564514+0000","last_clean":"2024-03-18T14:41:40.564514+0000","last_became_active":"2024-03-18T14:41:35.549713+0000","last_became_peered":"2024-03-18T14:41:35.549713+0000","last_unstale":"2024-03-18T14:41:40.564514+0000","last_undegraded":"2024-03-18T14:41:40.564514+0000","last_fullsized":"2024-03-18T14:41:40.564514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559089+0000","last_change":"2024-03-18T14:41:35.540097+0000","last_active":"2024-03-18T14:41:40.559089+0000","last_peered":"2024-03-18T14:41:40.559089+0000","last_clean":"2024-03-18T14:41:40.559089+0000","last_became_active":"2024-03-18T14:41:35.539823+0000","last_became_peered":"2024-03-18T14:41:35.539823+0000","last_unstale":"2024-03-18T14:41:40.559089+0000","last_undegraded":"2024-03-18T14:41:40.559089+0000","last_fullsized":"2024-03-18T14:41:40.559089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:59.228318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563495+0000","last_change":"2024-03-18T14:41:35.544547+0000","last_active":"2024-03-18T14:41:40.563495+0000","last_peered":"2024-03-18T14:41:40.563495+0000","last_clean":"2024-03-18T14:41:40.563495+0000","last_became_active":"2024-03-18T14:41:35.544026+0000","last_became_peered":"2024-03-18T14:41:35.544026+0000","last_unstale":"2024-03-18T14:41:40.563495+0000","last_undegraded":"2024-03-18T14:41:40.563495+0000","last_fullsized":"2024-03-18T14:41:40.563495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559538+0000","last_change":"2024-03-18T14:41:35.545773+0000","last_active":"2024-03-18T14:41:40.559538+0000","last_peered":"2024-03-18T14:41:40.559538+0000","last_clean":"2024-03-18T14:41:40.559538+0000","last_became_active":"2024-03-18T14:41:35.545449+0000","last_became_peered":"2024-03-18T14:41:35.545449+0000","last_unstale":"2024-03-18T14:41:40.559538+0000","last_undegraded":"2024-03-18T14:41:40.559538+0000","last_fullsized":"2024-03-18T14:41:40.559538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778786+0000","last_change":"2024-03-18T14:41:35.563339+0000","last_active":"2024-03-18T14:41:40.778786+0000","last_peered":"2024-03-18T14:41:40.778786+0000","last_clean":"2024-03-18T14:41:40.778786+0000","last_became_active":"2024-03-18T14:41:35.563117+0000","last_became_peered":"2024-03-18T14:41:35.563117+0000","last_unstale":"2024-03-18T14:41:40.778786+0000","last_undegraded":"2024-03-18T14:41:40.778786+0000","last_fullsized":"2024-03-18T14:41:40.778786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778522+0000","last_change":"2024-03-18T14:41:35.559014+0000","last_active":"2024-03-18T14:41:40.778522+0000","last_peered":"2024-03-18T14:41:40.778522+0000","last_clean":"2024-03-18T14:41:40.778522+0000","last_became_active":"2024-03-18T14:41:35.558349+0000","last_became_peered":"2024-03-18T14:41:35.558349+0000","last_unstale":"2024-03-18T14:41:40.778522+0000","last_undegraded":"2024-03-18T14:41:40.778522+0000","last_fullsized":"2024-03-18T14:41:40.778522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:11.086308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778971+0000","last_change":"2024-03-18T14:41:35.545841+0000","last_active":"2024-03-18T14:41:40.778971+0000","last_peered":"2024-03-18T14:41:40.778971+0000","last_clean":"2024-03-18T14:41:40.778971+0000","last_became_active":"2024-03-18T14:41:35.545543+0000","last_became_peered":"2024-03-18T14:41:35.545543+0000","last_unstale":"2024-03-18T14:41:40.778971+0000","last_undegraded":"2024-03-18T14:41:40.778971+0000","last_fullsized":"2024-03-18T14:41:40.778971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:37.204770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778782+0000","last_change":"2024-03-18T14:41:35.551018+0000","last_active":"2024-03-18T14:41:40.778782+0000","last_peered":"2024-03-18T14:41:40.778782+0000","last_clean":"2024-03-18T14:41:40.778782+0000","last_became_active":"2024-03-18T14:41:35.550737+0000","last_became_peered":"2024-03-18T14:41:35.550737+0000","last_unstale":"2024-03-18T14:41:40.778782+0000","last_undegraded":"2024-03-18T14:41:40.778782+0000","last_fullsized":"2024-03-18T14:41:40.778782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.560481+0000","last_change":"2024-03-18T14:41:35.547133+0000","last_active":"2024-03-18T14:41:40.560481+0000","last_peered":"2024-03-18T14:41:40.560481+0000","last_clean":"2024-03-18T14:41:40.560481+0000","last_became_active":"2024-03-18T14:41:35.546778+0000","last_became_peered":"2024-03-18T14:41:35.546778+0000","last_unstale":"2024-03-18T14:41:40.560481+0000","last_undegraded":"2024-03-18T14:41:40.560481+0000","last_fullsized":"2024-03-18T14:41:40.560481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:07.152984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559067+0000","last_change":"2024-03-18T14:41:35.550366+0000","last_active":"2024-03-18T14:41:40.559067+0000","last_peered":"2024-03-18T14:41:40.559067+0000","last_clean":"2024-03-18T14:41:40.559067+0000","last_became_active":"2024-03-18T14:41:35.550111+0000","last_became_peered":"2024-03-18T14:41:35.550111+0000","last_unstale":"2024-03-18T14:41:40.559067+0000","last_undegraded":"2024-03-18T14:41:40.559067+0000","last_fullsized":"2024-03-18T14:41:40.559067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:04.353967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778058+0000","last_change":"2024-03-18T14:41:35.536085+0000","last_active":"2024-03-18T14:41:40.778058+0000","last_peered":"2024-03-18T14:41:40.778058+0000","last_clean":"2024-03-18T14:41:40.778058+0000","last_became_active":"2024-03-18T14:41:35.535769+0000","last_became_peered":"2024-03-18T14:41:35.535769+0000","last_unstale":"2024-03-18T14:41:40.778058+0000","last_undegraded":"2024-03-18T14:41:40.778058+0000","last_fullsized":"2024-03-18T14:41:40.778058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:22.970440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778571+0000","last_change":"2024-03-18T14:41:35.550291+0000","last_active":"2024-03-18T14:41:40.778571+0000","last_peered":"2024-03-18T14:41:40.778571+0000","last_clean":"2024-03-18T14:41:40.778571+0000","last_became_active":"2024-03-18T14:41:35.549994+0000","last_became_peered":"2024-03-18T14:41:35.549994+0000","last_unstale":"2024-03-18T14:41:40.778571+0000","last_undegraded":"2024-03-18T14:41:40.778571+0000","last_fullsized":"2024-03-18T14:41:40.778571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:20.446494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563821+0000","last_change":"2024-03-18T14:41:35.551464+0000","last_active":"2024-03-18T14:41:40.563821+0000","last_peered":"2024-03-18T14:41:40.563821+0000","last_clean":"2024-03-18T14:41:40.563821+0000","last_became_active":"2024-03-18T14:41:35.551200+0000","last_became_peered":"2024-03-18T14:41:35.551200+0000","last_unstale":"2024-03-18T14:41:40.563821+0000","last_undegraded":"2024-03-18T14:41:40.563821+0000","last_fullsized":"2024-03-18T14:41:40.563821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559869+0000","last_change":"2024-03-18T14:41:35.558583+0000","last_active":"2024-03-18T14:41:40.559869+0000","last_peered":"2024-03-18T14:41:40.559869+0000","last_clean":"2024-03-18T14:41:40.559869+0000","last_became_active":"2024-03-18T14:41:35.557388+0000","last_became_peered":"2024-03-18T14:41:35.557388+0000","last_unstale":"2024-03-18T14:41:40.559869+0000","last_undegraded":"2024-03-18T14:41:40.559869+0000","last_fullsized":"2024-03-18T14:41:40.559869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:37.509816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778685+0000","last_change":"2024-03-18T14:41:35.556362+0000","last_active":"2024-03-18T14:41:40.778685+0000","last_peered":"2024-03-18T14:41:40.778685+0000","last_clean":"2024-03-18T14:41:40.778685+0000","last_became_active":"2024-03-18T14:41:35.556073+0000","last_became_peered":"2024-03-18T14:41:35.556073+0000","last_unstale":"2024-03-18T14:41:40.778685+0000","last_undegraded":"2024-03-18T14:41:40.778685+0000","last_fullsized":"2024-03-18T14:41:40.778685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:25.284296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778827+0000","last_change":"2024-03-18T14:41:35.553776+0000","last_active":"2024-03-18T14:41:40.778827+0000","last_peered":"2024-03-18T14:41:40.778827+0000","last_clean":"2024-03-18T14:41:40.778827+0000","last_became_active":"2024-03-18T14:41:35.553413+0000","last_became_peered":"2024-03-18T14:41:35.553413+0000","last_unstale":"2024-03-18T14:41:40.778827+0000","last_undegraded":"2024-03-18T14:41:40.778827+0000","last_fullsized":"2024-03-18T14:41:40.778827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:25:43.404692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778851+0000","last_change":"2024-03-18T14:41:35.556025+0000","last_active":"2024-03-18T14:41:40.778851+0000","last_peered":"2024-03-18T14:41:40.778851+0000","last_clean":"2024-03-18T14:41:40.778851+0000","last_became_active":"2024-03-18T14:41:35.555322+0000","last_became_peered":"2024-03-18T14:41:35.555322+0000","last_unstale":"2024-03-18T14:41:40.778851+0000","last_undegraded":"2024-03-18T14:41:40.778851+0000","last_fullsized":"2024-03-18T14:41:40.778851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:43.985250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.563141+0000","last_change":"2024-03-18T14:41:35.545290+0000","last_active":"2024-03-18T14:41:40.563141+0000","last_peered":"2024-03-18T14:41:40.563141+0000","last_clean":"2024-03-18T14:41:40.563141+0000","last_became_active":"2024-03-18T14:41:35.543962+0000","last_became_peered":"2024-03-18T14:41:35.543962+0000","last_unstale":"2024-03-18T14:41:40.563141+0000","last_undegraded":"2024-03-18T14:41:40.563141+0000","last_fullsized":"2024-03-18T14:41:40.563141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:54.790152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778288+0000","last_change":"2024-03-18T14:41:35.556987+0000","last_active":"2024-03-18T14:41:40.778288+0000","last_peered":"2024-03-18T14:41:40.778288+0000","last_clean":"2024-03-18T14:41:40.778288+0000","last_became_active":"2024-03-18T14:41:35.556713+0000","last_became_peered":"2024-03-18T14:41:35.556713+0000","last_unstale":"2024-03-18T14:41:40.778288+0000","last_undegraded":"2024-03-18T14:41:40.778288+0000","last_fullsized":"2024-03-18T14:41:40.778288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559461+0000","last_change":"2024-03-18T14:41:35.548724+0000","last_active":"2024-03-18T14:41:40.559461+0000","last_peered":"2024-03-18T14:41:40.559461+0000","last_clean":"2024-03-18T14:41:40.559461+0000","last_became_active":"2024-03-18T14:41:35.547693+0000","last_became_peered":"2024-03-18T14:41:35.547693+0000","last_unstale":"2024-03-18T14:41:40.559461+0000","last_undegraded":"2024-03-18T14:41:40.559461+0000","last_fullsized":"2024-03-18T14:41:40.559461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:06:49.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778449+0000","last_change":"2024-03-18T14:41:35.544193+0000","last_active":"2024-03-18T14:41:40.778449+0000","last_peered":"2024-03-18T14:41:40.778449+0000","last_clean":"2024-03-18T14:41:40.778449+0000","last_became_active":"2024-03-18T14:41:35.543909+0000","last_became_peered":"2024-03-18T14:41:35.543909+0000","last_unstale":"2024-03-18T14:41:40.778449+0000","last_undegraded":"2024-03-18T14:41:40.778449+0000","last_fullsized":"2024-03-18T14:41:40.778449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:46.086665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.559445+0000","last_change":"2024-03-18T14:41:35.549929+0000","last_active":"2024-03-18T14:41:40.559445+0000","last_peered":"2024-03-18T14:41:40.559445+0000","last_clean":"2024-03-18T14:41:40.559445+0000","last_became_active":"2024-03-18T14:41:35.547332+0000","last_became_peered":"2024-03-18T14:41:35.547332+0000","last_unstale":"2024-03-18T14:41:40.559445+0000","last_undegraded":"2024-03-18T14:41:40.559445+0000","last_fullsized":"2024-03-18T14:41:40.559445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:58:59.974945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.779000+0000","last_change":"2024-03-18T14:41:35.550445+0000","last_active":"2024-03-18T14:41:40.779000+0000","last_peered":"2024-03-18T14:41:40.779000+0000","last_clean":"2024-03-18T14:41:40.779000+0000","last_became_active":"2024-03-18T14:41:35.550120+0000","last_became_peered":"2024-03-18T14:41:35.550120+0000","last_unstale":"2024-03-18T14:41:40.779000+0000","last_undegraded":"2024-03-18T14:41:40.779000+0000","last_fullsized":"2024-03-18T14:41:40.779000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778964+0000","last_change":"2024-03-18T14:41:35.548653+0000","last_active":"2024-03-18T14:41:40.778964+0000","last_peered":"2024-03-18T14:41:40.778964+0000","last_clean":"2024-03-18T14:41:40.778964+0000","last_became_active":"2024-03-18T14:41:35.548305+0000","last_became_peered":"2024-03-18T14:41:35.548305+0000","last_unstale":"2024-03-18T14:41:40.778964+0000","last_undegraded":"2024-03-18T14:41:40.778964+0000","last_fullsized":"2024-03-18T14:41:40.778964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:25.002974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T14:41:40.778662+0000","last_change":"2024-03-18T14:41:35.560778+0000","last_active":"2024-03-18T14:41:40.778662+0000","last_peered":"2024-03-18T14:41:40.778662+0000","last_clean":"2024-03-18T14:41:40.778662+0000","last_became_active":"2024-03-18T14:41:35.560478+0000","last_became_peered":"2024-03-18T14:41:35.560478+0000","last_unstale":"2024-03-18T14:41:40.778662+0000","last_undegraded":"2024-03-18T14:41:40.778662+0000","last_fullsized":"2024-03-18T14:41:40.778662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:34.509969+0000","last_clean_scrub_stamp":"2024-03-18T14:41:34.509969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:36.745794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:42:03.385 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-18T14:42:03.385 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T14:42:03.385 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-18T14:42:03.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-18T14:42:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.520+0000 7f758d969640 1 Processor -- start 2024-03-18T14:42:03.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.520+0000 7f758d969640 1 -- start start 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f758d969640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75880ced00 0x7f7588162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f758d969640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75880d3210 0x7f75880d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f758d969640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75880b9660 con 0x7f758815e020 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f758d969640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f75880d2270 con 0x7f75880ced00 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f758d969640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f75880ce390 con 0x7f75880d3210 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75867fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75880ced00 0x7f7588162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75867fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75880ced00 0x7f7588162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39850/0 (socket says 172.21.15.6:39850) 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75877fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75880d3210 0x7f75880d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75867fc640 1 -- 172.21.15.6:0/4242589682 learned_addr learned my addr 172.21.15.6:0/4242589682 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f7586ffd640 1 --1- 172.21.15.6:0/4242589682 >> v1:172.21.15.92:6789/0 conn(0x7f758815e020 0x7f75880d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:59058/0 (socket says 172.21.15.6:59058) 2024-03-18T14:42:03.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75867fc640 1 -- 172.21.15.6:0/4242589682 >> v2:172.21.15.92:3300/0 conn(0x7f75880d3210 msgr2=0x7f75880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:03.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75867fc640 1 --2- 172.21.15.6:0/4242589682 >> v2:172.21.15.92:3300/0 conn(0x7f75880d3210 0x7f75880d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75867fc640 1 -- 172.21.15.6:0/4242589682 >> v1:172.21.15.92:6789/0 conn(0x7f758815e020 legacy=0x7f75880d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.521+0000 7f75867fc640 1 -- 172.21.15.6:0/4242589682 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75880b9900 con 0x7f75880ced00 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.522+0000 7f75867fc640 1 --2- 172.21.15.6:0/4242589682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880ced00 0x7f7588162380 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f758815e400 tx=0x7f757c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c6dde9870c4e89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.522+0000 7f7585ffb640 1 -- 172.21.15.6:0/4242589682 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c035020 con 0x7f75880ced00 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.522+0000 7f7585ffb640 1 -- 172.21.15.6:0/4242589682 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f757c031870 con 0x7f75880ced00 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.522+0000 7f7585ffb640 1 -- 172.21.15.6:0/4242589682 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c031c40 con 0x7f75880ced00 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 -- 172.21.15.6:0/4242589682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880ced00 msgr2=0x7f7588162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 --2- 172.21.15.6:0/4242589682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880ced00 0x7f7588162380 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f758815e400 tx=0x7f757c02b910 comp rx=0 tx=0).stop 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 -- 172.21.15.6:0/4242589682 shutdown_connections 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 --2- 172.21.15.6:0/4242589682 >> v2:172.21.15.92:3300/0 conn(0x7f75880d3210 0x7f75880d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 --2- 172.21.15.6:0/4242589682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880ced00 0x7f7588162380 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 -- 172.21.15.6:0/4242589682 >> 172.21.15.6:0/4242589682 conn(0x7f7588072ce0 msgr2=0x7f75880730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 -- 172.21.15.6:0/4242589682 shutdown_connections 2024-03-18T14:42:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.523+0000 7f758d969640 1 -- 172.21.15.6:0/4242589682 wait complete. 2024-03-18T14:42:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.524+0000 7f758d969640 1 Processor -- start 2024-03-18T14:42:03.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.524+0000 7f758d969640 1 -- start start 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f758d969640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75880ced00 0x7f75880b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f758d969640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880d3210 0x7f758815f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f758d969640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758815e020 0x7f7588144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f758d969640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f75880cf2c0 con 0x7f75880d3210 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f758d969640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75880ce890 con 0x7f75880ced00 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f758d969640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f75880cf440 con 0x7f758815e020 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880d3210 0x7f758815f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880d3210 0x7f758815f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39860/0 (socket says 172.21.15.6:39860) 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 -- 172.21.15.6:0/2785876900 learned_addr learned my addr 172.21.15.6:0/2785876900 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f7586ffd640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75880ced00 0x7f75880b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75877fe640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758815e020 0x7f7588144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 -- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758815e020 msgr2=0x7f7588144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758815e020 0x7f7588144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 -- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75880ced00 msgr2=0x7f75880b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75880ced00 0x7f75880b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 -- 172.21.15.6:0/2785876900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75880b3b40 con 0x7f75880d3210 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.525+0000 7f75867fc640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880d3210 0x7f758815f740 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f757c02b950 tx=0x7f757c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.526+0000 7f756ffff640 1 -- 172.21.15.6:0/2785876900 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c040070 con 0x7f75880d3210 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.526+0000 7f756ffff640 1 -- 172.21.15.6:0/2785876900 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f757c030d00 con 0x7f75880d3210 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.526+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7588143db0 con 0x7f75880d3210 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.526+0000 7f756ffff640 1 -- 172.21.15.6:0/2785876900 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c035040 con 0x7f75880d3210 2024-03-18T14:42:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.526+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7588144350 con 0x7f75880d3210 2024-03-18T14:42:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.527+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f758808afe0 con 0x7f75880d3210 2024-03-18T14:42:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.527+0000 7f756ffff640 1 -- 172.21.15.6:0/2785876900 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f757c038020 con 0x7f75880d3210 2024-03-18T14:42:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.527+0000 7f756ffff640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f757406bbd0 0x7f757406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.527+0000 7f756ffff640 1 -- 172.21.15.6:0/2785876900 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f757c0aca90 con 0x7f75880d3210 2024-03-18T14:42:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.527+0000 7f7586ffd640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f757406bbd0 0x7f757406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.528+0000 7f7586ffd640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f757406bbd0 0x7f757406e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f75800012e0 tx=0x7f7580001610 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.532+0000 7f756ffff640 1 -- 172.21.15.6:0/2785876900 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f757c07abf0 con 0x7f75880d3210 2024-03-18T14:42:03.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.760+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f758804f620 con 0x7f75880d3210 2024-03-18T14:42:03.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.760+0000 7f756ffff640 1 -- 172.21.15.6:0/2785876900 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f757c07ec90 con 0x7f75880d3210 2024-03-18T14:42:03.762 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:03.762 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T14:42:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f757406bbd0 msgr2=0x7f757406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f757406bbd0 0x7f757406e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f75800012e0 tx=0x7f7580001610 comp rx=0 tx=0).stop 2024-03-18T14:42:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880d3210 msgr2=0x7f758815f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880d3210 0x7f758815f740 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f757c02b950 tx=0x7f757c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 shutdown_connections 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f757406bbd0 0x7f757406e090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758815e020 0x7f7588144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75880d3210 0x7f758815f740 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 --2- 172.21.15.6:0/2785876900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75880ced00 0x7f75880b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 >> 172.21.15.6:0/2785876900 conn(0x7f7588072ce0 msgr2=0x7f75880cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 shutdown_connections 2024-03-18T14:42:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.762+0000 7f758d969640 1 -- 172.21.15.6:0/2785876900 wait complete. 2024-03-18T14:42:03.775 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-18T14:42:03.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:03.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.875+0000 7fdebcedc640 1 Processor -- start 2024-03-18T14:42:03.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.875+0000 7fdebcedc640 1 -- start start 2024-03-18T14:42:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.875+0000 7fdebcedc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdeb815e020 0x7fdeb8162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdebcedc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb80b9660 con 0x7fdeb80ced00 2024-03-18T14:42:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdebcedc640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb80ce390 con 0x7fdeb80d3210 2024-03-18T14:42:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdebcedc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb80d2270 con 0x7fdeb815e020 2024-03-18T14:42:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeaffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdeb80ced00 0x7fdeb80d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38546/0 (socket says 172.21.15.6:38546) 2024-03-18T14:42:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeaffff640 1 -- 172.21.15.6:0/3889315039 learned_addr learned my addr 172.21.15.6:0/3889315039 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb659c640 1 --2- 172.21.15.6:0/3889315039 >> v2:172.21.15.92:3300/0 conn(0x7fdeb815e020 0x7fdeb8162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb5d9b640 1 -- 172.21.15.6:0/3889315039 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806954548 0 0) 0x7fdeb80b9660 con 0x7fdeb80ced00 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb5d9b640 1 -- 172.21.15.6:0/3889315039 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea4003520 con 0x7fdeb80ced00 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb5d9b640 1 -- 172.21.15.6:0/3889315039 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309280424 0 0) 0x7fdeb80ce390 con 0x7fdeb80d3210 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb5d9b640 1 -- 172.21.15.6:0/3889315039 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdeb80b9660 con 0x7fdeb80d3210 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb659c640 1 -- 172.21.15.6:0/3889315039 >> v1:172.21.15.92:6789/0 conn(0x7fdeb80d3210 legacy=0x7fdeb80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb659c640 1 -- 172.21.15.6:0/3889315039 >> v1:172.21.15.6:6789/0 conn(0x7fdeb80ced00 legacy=0x7fdeb80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.876+0000 7fdeb659c640 1 -- 172.21.15.6:0/3889315039 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb80b9900 con 0x7fdeb815e020 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.877+0000 7fdeb659c640 1 --2- 172.21.15.6:0/3889315039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 0x7fdeb8162380 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d047c0a86c575057 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.877+0000 7fdeb5d9b640 1 -- 172.21.15.6:0/3889315039 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8035020 con 0x7fdeb815e020 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.877+0000 7fdeb5d9b640 1 -- 172.21.15.6:0/3889315039 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea8031830 con 0x7fdeb815e020 2024-03-18T14:42:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.877+0000 7fdeb5d9b640 1 -- 172.21.15.6:0/3889315039 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8031be0 con 0x7fdeb815e020 2024-03-18T14:42:03.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.878+0000 7fdebcedc640 1 -- 172.21.15.6:0/3889315039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 msgr2=0x7fdeb8162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.878+0000 7fdebcedc640 1 --2- 172.21.15.6:0/3889315039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 0x7fdeb8162380 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.878+0000 7fdebcedc640 1 -- 172.21.15.6:0/3889315039 shutdown_connections 2024-03-18T14:42:03.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.878+0000 7fdebcedc640 1 --2- 172.21.15.6:0/3889315039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 0x7fdeb8162380 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.878+0000 7fdebcedc640 1 -- 172.21.15.6:0/3889315039 >> 172.21.15.6:0/3889315039 conn(0x7fdeb8072ce0 msgr2=0x7fdeb80730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:03.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.878+0000 7fdebcedc640 1 -- 172.21.15.6:0/3889315039 shutdown_connections 2024-03-18T14:42:03.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.878+0000 7fdebcedc640 1 -- 172.21.15.6:0/3889315039 wait complete. 2024-03-18T14:42:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.879+0000 7fdebcedc640 1 Processor -- start 2024-03-18T14:42:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.879+0000 7fdebcedc640 1 -- start start 2024-03-18T14:42:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdebcedc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdeb80ced00 0x7fdeb815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdebcedc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdeb80d3210 0x7fdeb815ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdebcedc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 0x7fdeb81604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdebcedc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb80cf2c0 con 0x7fdeb80d3210 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdebcedc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb80ce890 con 0x7fdeb815e020 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdebcedc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb80cf440 con 0x7fdeb80ced00 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdeb80d3210 0x7fdeb815ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdeb80d3210 0x7fdeb815ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39864/0 (socket says 172.21.15.6:39864) 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 -- 172.21.15.6:0/298847984 learned_addr learned my addr 172.21.15.6:0/298847984 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeb659c640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdeb80ced00 0x7fdeb815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeb6d9d640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 0x7fdeb81604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 -- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdeb80ced00 msgr2=0x7fdeb815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdeb80ced00 0x7fdeb815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 -- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 msgr2=0x7fdeb81604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 0x7fdeb81604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 -- 172.21.15.6:0/298847984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb815f520 con 0x7fdeb80d3210 2024-03-18T14:42:03.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaffff640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdeb80d3210 0x7fdeb815ffb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdea000c4c0 tx=0x7fdea000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaeffd640 1 -- 172.21.15.6:0/298847984 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea0017070 con 0x7fdeb80d3210 2024-03-18T14:42:03.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaeffd640 1 -- 172.21.15.6:0/298847984 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea0005ce0 con 0x7fdeb80d3210 2024-03-18T14:42:03.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb815f800 con 0x7fdeb80d3210 2024-03-18T14:42:03.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.880+0000 7fdeaeffd640 1 -- 172.21.15.6:0/298847984 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea0007270 con 0x7fdeb80d3210 2024-03-18T14:42:03.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.881+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb8153310 con 0x7fdeb80d3210 2024-03-18T14:42:03.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.882+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb808afe0 con 0x7fdeb80d3210 2024-03-18T14:42:03.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.882+0000 7fdeaeffd640 1 -- 172.21.15.6:0/298847984 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdea001a070 con 0x7fdeb80d3210 2024-03-18T14:42:03.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.882+0000 7fdeaeffd640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde9806bbd0 0x7fde9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:03.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.882+0000 7fdeaeffd640 1 -- 172.21.15.6:0/298847984 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdea008a040 con 0x7fdeb80d3210 2024-03-18T14:42:03.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.882+0000 7fdeb659c640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde9806bbd0 0x7fde9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:03.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.883+0000 7fdeb659c640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde9806bbd0 0x7fde9806e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdea8036000 tx=0x7fdea803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:03.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:03.887+0000 7fdeaeffd640 1 -- 172.21.15.6:0/298847984 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdea00581b0 con 0x7fdeb80d3210 2024-03-18T14:42:04.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.069+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdeb8156520 con 0x7fdeb80d3210 2024-03-18T14:42:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.070+0000 7fdeaeffd640 1 -- 172.21.15.6:0/298847984 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7fdea005c250 con 0x7fdeb80d3210 2024-03-18T14:42:04.072 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:04.072 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde9806bbd0 msgr2=0x7fde9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde9806bbd0 0x7fde9806e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdea8036000 tx=0x7fdea803c040 comp rx=0 tx=0).stop 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdeb80d3210 msgr2=0x7fdeb815ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdeb80d3210 0x7fdeb815ffb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdea000c4c0 tx=0x7fdea000c980 comp rx=0 tx=0).stop 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 shutdown_connections 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde9806bbd0 0x7fde9806e090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdeb815e020 0x7fdeb81604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdeb80d3210 0x7fdeb815ffb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 --2- 172.21.15.6:0/298847984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdeb80ced00 0x7fdeb815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 >> 172.21.15.6:0/298847984 conn(0x7fdeb8072ce0 msgr2=0x7fdeb80d3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 shutdown_connections 2024-03-18T14:42:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.072+0000 7fdebcedc640 1 -- 172.21.15.6:0/298847984 wait complete. 2024-03-18T14:42:04.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:04.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:04.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.187+0000 7f6864697640 1 Processor -- start 2024-03-18T14:42:04.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.187+0000 7f6864697640 1 -- start start 2024-03-18T14:42:04.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6864697640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f685c0ce9d0 0x7f685c0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6864697640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6864697640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685c0d29a0 con 0x7f685c0d2510 2024-03-18T14:42:04.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6864697640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f685c0cd8d0 con 0x7f685c0ce9d0 2024-03-18T14:42:04.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6864697640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f685c0cfac0 con 0x7f685c0d0730 2024-03-18T14:42:04.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6861c32640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f685c0d2510 0x7f685c0d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38556/0 (socket says 172.21.15.6:38556) 2024-03-18T14:42:04.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6861c32640 1 -- 172.21.15.6:0/2728947296 learned_addr learned my addr 172.21.15.6:0/2728947296 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6862c34640 1 --2- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.188+0000 7f6862433640 1 --2- 172.21.15.6:0/2728947296 >> v2:172.21.15.92:3300/0 conn(0x7f685c0ce9d0 0x7f685c0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6861431640 1 -- 172.21.15.6:0/2728947296 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042729691 0 0) 0x7f685c0d29a0 con 0x7f685c0d2510 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6861431640 1 -- 172.21.15.6:0/2728947296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6844003520 con 0x7f685c0d2510 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6862433640 1 -- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 msgr2=0x7f685c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6862433640 1 --2- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6862433640 1 -- 172.21.15.6:0/2728947296 >> v1:172.21.15.6:6789/0 conn(0x7f685c0d2510 legacy=0x7f685c0d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6862433640 1 -- 172.21.15.6:0/2728947296 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c0cccc0 con 0x7f685c0ce9d0 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6862c34640 1 --2- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.189+0000 7f6862433640 1 --2- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 0x7f685c0cc650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f684c009790 tx=0x7f684c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ff89d965682e9a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6861431640 1 -- 172.21.15.6:0/2728947296 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c031770 con 0x7f685c0ce9d0 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6861431640 1 -- 172.21.15.6:0/2728947296 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f684c037070 con 0x7f685c0ce9d0 2024-03-18T14:42:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6861431640 1 -- 172.21.15.6:0/2728947296 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c031b30 con 0x7f685c0ce9d0 2024-03-18T14:42:04.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 -- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 msgr2=0x7f685c0cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 --2- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 0x7f685c0cc650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f684c009790 tx=0x7f684c02b810 comp rx=0 tx=0).stop 2024-03-18T14:42:04.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 -- 172.21.15.6:0/2728947296 shutdown_connections 2024-03-18T14:42:04.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 --2- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 --2- 172.21.15.6:0/2728947296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 0x7f685c0cc650 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 -- 172.21.15.6:0/2728947296 >> 172.21.15.6:0/2728947296 conn(0x7f685c08b0c0 msgr2=0x7f685c0d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:04.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 -- 172.21.15.6:0/2728947296 shutdown_connections 2024-03-18T14:42:04.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.190+0000 7f6864697640 1 -- 172.21.15.6:0/2728947296 wait complete. 2024-03-18T14:42:04.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.191+0000 7f6864697640 1 Processor -- start 2024-03-18T14:42:04.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.191+0000 7f6864697640 1 -- start start 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6864697640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 0x7f685c1f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6864697640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6864697640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f685c0d2510 0x7f685c156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6864697640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f685c0cd560 con 0x7f685c0d2510 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6864697640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f685c0cd280 con 0x7f685c0ce9d0 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6864697640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f685c0cedb0 con 0x7f685c0d0730 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6862c34640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f685c0d2510 0x7f685c156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6862c34640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f685c0d2510 0x7f685c156510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39866/0 (socket says 172.21.15.6:39866) 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6862c34640 1 -- 172.21.15.6:0/3297860939 learned_addr learned my addr 172.21.15.6:0/3297860939 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6861c32640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.192+0000 7f6862433640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 0x7f685c1f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6862c34640 1 -- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 msgr2=0x7f685c155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6862c34640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6862c34640 1 -- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 msgr2=0x7f685c1f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6862c34640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 0x7f685c1f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6862c34640 1 -- 172.21.15.6:0/3297860939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c0c74f0 con 0x7f685c0d2510 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6862c34640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f685c0d2510 0x7f685c156510 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f685400efc0 tx=0x7f685400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f685affd640 1 -- 172.21.15.6:0/3297860939 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854018020 con 0x7f685c0d2510 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f685affd640 1 -- 172.21.15.6:0/3297860939 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6854014480 con 0x7f685c0d2510 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685c0c77d0 con 0x7f685c0d2510 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f685affd640 1 -- 172.21.15.6:0/3297860939 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854014850 con 0x7f685c0d2510 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.193+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685c0c7d90 con 0x7f685c0d2510 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.194+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f685c0751b0 con 0x7f685c0d2510 2024-03-18T14:42:04.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.194+0000 7f685affd640 1 -- 172.21.15.6:0/3297860939 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6854005d10 con 0x7f685c0d2510 2024-03-18T14:42:04.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.195+0000 7f685affd640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f684806bbd0 0x7f684806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.195+0000 7f685affd640 1 -- 172.21.15.6:0/3297860939 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f685408b9d0 con 0x7f685c0d2510 2024-03-18T14:42:04.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.195+0000 7f6862433640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f684806bbd0 0x7f684806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.195+0000 7f6862433640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f684806bbd0 0x7f684806e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f684c036000 tx=0x7f684c03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.199+0000 7f685affd640 1 -- 172.21.15.6:0/3297860939 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6854022030 con 0x7f685c0d2510 2024-03-18T14:42:04.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.378+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f685c0cfe00 con 0x7f685c0d2510 2024-03-18T14:42:04.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.379+0000 7f685affd640 1 -- 172.21.15.6:0/3297860939 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f6854010040 con 0x7f685c0d2510 2024-03-18T14:42:04.380 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f684806bbd0 msgr2=0x7f684806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f684806bbd0 0x7f684806e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f684c036000 tx=0x7f684c03d040 comp rx=0 tx=0).stop 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f685c0d2510 msgr2=0x7f685c156510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f685c0d2510 0x7f685c156510 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f685400efc0 tx=0x7f685400c490 comp rx=0 tx=0).stop 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 shutdown_connections 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f684806bbd0 0x7f684806e090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f685c0d2510 0x7f685c156510 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f685c0d0730 0x7f685c155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 --2- 172.21.15.6:0/3297860939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c0ce9d0 0x7f685c1f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 >> 172.21.15.6:0/3297860939 conn(0x7f685c08b0c0 msgr2=0x7f685c0b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 shutdown_connections 2024-03-18T14:42:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.381+0000 7f6864697640 1 -- 172.21.15.6:0/3297860939 wait complete. 2024-03-18T14:42:04.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:04.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.497+0000 7f1a844dc640 1 Processor -- start 2024-03-18T14:42:04.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.497+0000 7f1a844dc640 1 -- start start 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a844dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c0b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a844dc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a7c0b5c80 0x7f1a7c0b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a844dc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a7c15e3b0 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a844dc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1a7c0b8470 con 0x7f1a7c0b5c80 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a844dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7c0b8750 con 0x7f1a7c0d2e50 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a82a79640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a7c0b5c80 0x7f1a7c0b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a82278640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1a7c15dfd0 0x7f1a7c0b9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38564/0 (socket says 172.21.15.6:38564) 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a82278640 1 -- 172.21.15.6:0/2410205804 learned_addr learned my addr 172.21.15.6:0/2410205804 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a81a77640 1 --2- 172.21.15.6:0/2410205804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c0b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a82a79640 1 -- 172.21.15.6:0/2410205804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 msgr2=0x7f1a7c0b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a82a79640 1 --2- 172.21.15.6:0/2410205804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c0b5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a82a79640 1 -- 172.21.15.6:0/2410205804 >> v1:172.21.15.6:6789/0 conn(0x7f1a7c15dfd0 legacy=0x7f1a7c0b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a82a79640 1 -- 172.21.15.6:0/2410205804 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c0b8a30 con 0x7f1a7c0b5c80 2024-03-18T14:42:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.498+0000 7f1a81276640 1 -- 172.21.15.6:0/2410205804 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588405211 0 0) 0x7f1a7c15e3b0 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.499+0000 7f1a82a79640 1 --2- 172.21.15.6:0/2410205804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c0b5c80 0x7f1a7c0b6df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009d90 tx=0x7f1a7402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645c8fdce0611a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.499+0000 7f1a81276640 1 -- 172.21.15.6:0/2410205804 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74039070 con 0x7f1a7c0b5c80 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.499+0000 7f1a81276640 1 -- 172.21.15.6:0/2410205804 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7402cc50 con 0x7f1a7c0b5c80 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.499+0000 7f1a81276640 1 -- 172.21.15.6:0/2410205804 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74034440 con 0x7f1a7c0b5c80 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 -- 172.21.15.6:0/2410205804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c0b5c80 msgr2=0x7f1a7c0b6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 --2- 172.21.15.6:0/2410205804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c0b5c80 0x7f1a7c0b6df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009d90 tx=0x7f1a7402bea0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 -- 172.21.15.6:0/2410205804 shutdown_connections 2024-03-18T14:42:04.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 --2- 172.21.15.6:0/2410205804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c0b5c80 0x7f1a7c0b6df0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 --2- 172.21.15.6:0/2410205804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c0b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 -- 172.21.15.6:0/2410205804 >> 172.21.15.6:0/2410205804 conn(0x7f1a7c072d10 msgr2=0x7f1a7c073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:04.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 -- 172.21.15.6:0/2410205804 shutdown_connections 2024-03-18T14:42:04.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.500+0000 7f1a844dc640 1 -- 172.21.15.6:0/2410205804 wait complete. 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.501+0000 7f1a844dc640 1 Processor -- start 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.501+0000 7f1a844dc640 1 -- start start 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a844dc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c0b5c80 0x7f1a7c0cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a844dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a844dc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c15dfd0 0x7f1a7c0d3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a844dc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c0b74b0 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a844dc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c0b7330 con 0x7f1a7c0b5c80 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a844dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7c0b7630 con 0x7f1a7c0d2e50 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c15dfd0 0x7f1a7c0d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c15dfd0 0x7f1a7c0d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39874/0 (socket says 172.21.15.6:39874) 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 -- 172.21.15.6:0/3862296273 learned_addr learned my addr 172.21.15.6:0/3862296273 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a81a77640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82278640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c0b5c80 0x7f1a7c0cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 -- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 msgr2=0x7f1a7c151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 -- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c0b5c80 msgr2=0x7f1a7c0cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c0b5c80 0x7f1a7c0cd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 -- 172.21.15.6:0/3862296273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c0aabc0 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a82a79640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c15dfd0 0x7f1a7c0d3310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009eb0 tx=0x7f1a74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a72ffd640 1 -- 172.21.15.6:0/3862296273 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74039050 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a72ffd640 1 -- 172.21.15.6:0/3862296273 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a74045070 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a72ffd640 1 -- 172.21.15.6:0/3862296273 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74009450 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.502+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a7c0aaea0 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.503+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a7c0ab460 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.504+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a7c0751b0 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.504+0000 7f1a72ffd640 1 -- 172.21.15.6:0/3862296273 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1a7403e610 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.504+0000 7f1a72ffd640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a6006bbd0 0x7f1a6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.504+0000 7f1a72ffd640 1 -- 172.21.15.6:0/3862296273 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1a740ad670 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.505+0000 7f1a82278640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a6006bbd0 0x7f1a6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.505+0000 7f1a82278640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a6006bbd0 0x7f1a6006e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1a78007fd0 tx=0x7f1a78007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.509+0000 7f1a72ffd640 1 -- 172.21.15.6:0/3862296273 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1a7407b860 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.691+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1a7c06b910 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.692+0000 7f1a72ffd640 1 -- 172.21.15.6:0/3862296273 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f1a7407f900 con 0x7f1a7c15dfd0 2024-03-18T14:42:04.695 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a6006bbd0 msgr2=0x7f1a6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a6006bbd0 0x7f1a6006e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1a78007fd0 tx=0x7f1a78007de0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c15dfd0 msgr2=0x7f1a7c0d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c15dfd0 0x7f1a7c0d3310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009eb0 tx=0x7f1a74004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 shutdown_connections 2024-03-18T14:42:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a6006bbd0 0x7f1a6006e090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a7c15dfd0 0x7f1a7c0d3310 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a7c0d2e50 0x7f1a7c151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 --2- 172.21.15.6:0/3862296273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c0b5c80 0x7f1a7c0cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 >> 172.21.15.6:0/3862296273 conn(0x7f1a7c072d10 msgr2=0x7f1a7c0b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 shutdown_connections 2024-03-18T14:42:04.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.694+0000 7f1a844dc640 1 -- 172.21.15.6:0/3862296273 wait complete. 2024-03-18T14:42:04.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:04.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.810+0000 7fe3fe1c4640 1 Processor -- start 2024-03-18T14:42:04.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.810+0000 7fe3fe1c4640 1 -- start start 2024-03-18T14:42:04.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3fe1c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3fe1c4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3f80b7e00 0x7fe3f80b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3fe1c4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3f815e3b0 con 0x7fe3f80b5740 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3fe1c4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe3f80b60e0 con 0x7fe3f80b7e00 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3fe1c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f80b8470 con 0x7fe3f815dfd0 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f77fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3f80b7e00 0x7fe3f80b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3effff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe3f80b5740 0x7fe3f80d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:59086/0 (socket says 172.21.15.6:59086) 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3f80b7e00 0x7fe3f80b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56702/0 (socket says 172.21.15.6:56702) 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3effff640 1 -- 172.21.15.6:0/4249453046 learned_addr learned my addr 172.21.15.6:0/4249453046 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f7fff640 1 -- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 msgr2=0x7fe3f80b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f7fff640 1 --2- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f7fff640 1 -- 172.21.15.6:0/4249453046 >> v1:172.21.15.92:6789/0 conn(0x7fe3f80b5740 legacy=0x7fe3f80d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f7fff640 1 -- 172.21.15.6:0/4249453046 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f80b8750 con 0x7fe3f80b7e00 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f6ffd640 1 -- 172.21.15.6:0/4249453046 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395749810 0 0) 0x7fe3f815e3b0 con 0x7fe3f80b5740 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.811+0000 7fe3f77fe640 1 --2- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.812+0000 7fe3f7fff640 1 --2- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f80b5ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009d90 tx=0x7fe3f002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d06136367adaf57d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.812+0000 7fe3f6ffd640 1 -- 172.21.15.6:0/4249453046 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0039070 con 0x7fe3f80b7e00 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.812+0000 7fe3f6ffd640 1 -- 172.21.15.6:0/4249453046 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f002cc50 con 0x7fe3f80b7e00 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.812+0000 7fe3f6ffd640 1 -- 172.21.15.6:0/4249453046 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0034440 con 0x7fe3f80b7e00 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 msgr2=0x7fe3f80b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f80b5ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009d90 tx=0x7fe3f002bea0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4249453046 shutdown_connections 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f80b5ba0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4249453046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4249453046 >> 172.21.15.6:0/4249453046 conn(0x7fe3f8072d10 msgr2=0x7fe3f8073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:04.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4249453046 shutdown_connections 2024-03-18T14:42:04.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.813+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4249453046 wait complete. 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.814+0000 7fe3fe1c4640 1 Processor -- start 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.814+0000 7fe3fe1c4640 1 -- start start 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3fe1c4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f80b5740 0x7fe3f815b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3fe1c4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3fe1c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80ab220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3fe1c4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f80b9b90 con 0x7fe3f80b5740 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3fe1c4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f80b8d10 con 0x7fe3f80b7e00 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3fe1c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f80d2550 con 0x7fe3f815dfd0 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3effff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f80b5740 0x7fe3f815b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f80b5740 0x7fe3f815b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39888/0 (socket says 172.21.15.6:39888) 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f77fe640 1 -- 172.21.15.6:0/4229915809 learned_addr learned my addr 172.21.15.6:0/4229915809 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80ab220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40388/0 (socket says 172.21.15.6:40388) 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3effff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f815ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56712/0 (socket says 172.21.15.6:56712) 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 -- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 msgr2=0x7fe3f815ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f815ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 -- 172.21.15.6:0/4229915809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f80b5740 msgr2=0x7fe3f815b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f80b5740 0x7fe3f815b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 -- 172.21.15.6:0/4229915809 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f80c7890 con 0x7fe3f815dfd0 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f77fe640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f80b5740 0x7fe3f815b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f7fff640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80ab220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0033000 tx=0x7fe3f0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f4ff9640 1 -- 172.21.15.6:0/4229915809 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0039050 con 0x7fe3f815dfd0 2024-03-18T14:42:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f4ff9640 1 -- 172.21.15.6:0/4229915809 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f0045070 con 0x7fe3f815dfd0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f80c7b70 con 0x7fe3f815dfd0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.815+0000 7fe3f4ff9640 1 -- 172.21.15.6:0/4229915809 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0009450 con 0x7fe3f815dfd0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.816+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f80c8130 con 0x7fe3f815dfd0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.816+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f80751b0 con 0x7fe3f815dfd0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.817+0000 7fe3f4ff9640 1 -- 172.21.15.6:0/4229915809 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe3f003e610 con 0x7fe3f815dfd0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.817+0000 7fe3f4ff9640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d806bc00 0x7fe3d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.817+0000 7fe3f4ff9640 1 -- 172.21.15.6:0/4229915809 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe3f00ad650 con 0x7fe3f815dfd0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.817+0000 7fe3f77fe640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d806bc00 0x7fe3d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.818+0000 7fe3f77fe640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d806bc00 0x7fe3d806e0c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3f81f6da0 tx=0x7fe3e8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:04.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:04.822+0000 7fe3f4ff9640 1 -- 172.21.15.6:0/4229915809 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe3f007b7c0 con 0x7fe3f815dfd0 2024-03-18T14:42:05.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.003+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe3f80731f0 con 0x7fe3f815dfd0 2024-03-18T14:42:05.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.003+0000 7fe3f4ff9640 1 -- 172.21.15.6:0/4229915809 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fe3f007f860 con 0x7fe3f815dfd0 2024-03-18T14:42:05.006 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d806bc00 msgr2=0x7fe3d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d806bc00 0x7fe3d806e0c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3f81f6da0 tx=0x7fe3e8009190 comp rx=0 tx=0).stop 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 msgr2=0x7fe3f80ab220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80ab220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0033000 tx=0x7fe3f0004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 shutdown_connections 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d806bc00 0x7fe3d806e0c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f815dfd0 0x7fe3f80ab220 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f80b7e00 0x7fe3f815ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 --2- 172.21.15.6:0/4229915809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f80b5740 0x7fe3f815b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 >> 172.21.15.6:0/4229915809 conn(0x7fe3f8072d10 msgr2=0x7fe3f80b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:05.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 shutdown_connections 2024-03-18T14:42:05.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.005+0000 7fe3fe1c4640 1 -- 172.21.15.6:0/4229915809 wait complete. 2024-03-18T14:42:05.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:05.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:42:05.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:42:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.121+0000 7f95e8f48640 1 Processor -- start 2024-03-18T14:42:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.121+0000 7f95e8f48640 1 -- start start 2024-03-18T14:42:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e8f48640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e40d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e8f48640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f95e40b6140 0x7f95e40ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e8f48640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e40b65a0 con 0x7f95e40d0450 2024-03-18T14:42:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e8f48640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f95e415e410 con 0x7f95e40b6140 2024-03-18T14:42:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e8f48640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f95e40ce300 con 0x7f95e415e030 2024-03-18T14:42:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e2d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f95e40d0450 0x7f95e40d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38572/0 (socket says 172.21.15.6:38572) 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e2d9d640 1 -- 172.21.15.6:0/370051898 learned_addr learned my addr 172.21.15.6:0/370051898 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e259c640 1 --2- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e40d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e1d9b640 1 --2- 172.21.15.6:0/370051898 >> v2:172.21.15.92:3300/0 conn(0x7f95e40b6140 0x7f95e40ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e159a640 1 -- 172.21.15.6:0/370051898 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3064743347 0 0) 0x7f95e40b65a0 con 0x7f95e40d0450 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e159a640 1 -- 172.21.15.6:0/370051898 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95d0003520 con 0x7f95e40d0450 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e1d9b640 1 -- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 msgr2=0x7f95e40d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e1d9b640 1 --2- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e40d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e1d9b640 1 -- 172.21.15.6:0/370051898 >> v1:172.21.15.6:6789/0 conn(0x7f95e40d0450 legacy=0x7f95e40d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.122+0000 7f95e1d9b640 1 -- 172.21.15.6:0/370051898 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e40d2fd0 con 0x7f95e40b6140 2024-03-18T14:42:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.123+0000 7f95e1d9b640 1 --2- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40ceec0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0097a0 tx=0x7f95cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bdb08dce2dbafe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.123+0000 7f95e159a640 1 -- 172.21.15.6:0/370051898 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc035020 con 0x7f95e40b6140 2024-03-18T14:42:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.123+0000 7f95e159a640 1 -- 172.21.15.6:0/370051898 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95cc031830 con 0x7f95e40b6140 2024-03-18T14:42:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.123+0000 7f95e159a640 1 -- 172.21.15.6:0/370051898 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc031c00 con 0x7f95e40b6140 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 -- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 msgr2=0x7f95e40ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 --2- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40ceec0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0097a0 tx=0x7f95cc02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 -- 172.21.15.6:0/370051898 shutdown_connections 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 --2- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40ceec0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 --2- 172.21.15.6:0/370051898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e40d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 -- 172.21.15.6:0/370051898 >> 172.21.15.6:0/370051898 conn(0x7f95e406c8c0 msgr2=0x7f95e406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 -- 172.21.15.6:0/370051898 shutdown_connections 2024-03-18T14:42:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.124+0000 7f95e8f48640 1 -- 172.21.15.6:0/370051898 wait complete. 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.125+0000 7f95e8f48640 1 Processor -- start 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.125+0000 7f95e8f48640 1 -- start start 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e8f48640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e8f48640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e40d0450 0x7f95e4159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e8f48640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e41586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e8f48640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f95e40ce830 con 0x7f95e40d0450 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e8f48640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f95e40ce550 con 0x7f95e40b6140 2024-03-18T14:42:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e8f48640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f95e40cf400 con 0x7f95e415e030 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e40d0450 0x7f95e4159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e40d0450 0x7f95e4159380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39898/0 (socket says 172.21.15.6:39898) 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 -- 172.21.15.6:0/4161497986 learned_addr learned my addr 172.21.15.6:0/4161497986 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e259c640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e2d9d640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e41586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 -- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 msgr2=0x7f95e41586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e41586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 -- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 msgr2=0x7f95e40cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e1d9b640 1 -- 172.21.15.6:0/4161497986 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e40cb790 con 0x7f95e40d0450 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.126+0000 7f95e259c640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.127+0000 7f95e1d9b640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e40d0450 0x7f95e4159380 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0098c0 tx=0x7f95cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.127+0000 7f95daffd640 1 -- 172.21.15.6:0/4161497986 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc040070 con 0x7f95e40d0450 2024-03-18T14:42:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.127+0000 7f95daffd640 1 -- 172.21.15.6:0/4161497986 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95cc03e070 con 0x7f95e40d0450 2024-03-18T14:42:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.127+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e4158c20 con 0x7f95e40d0450 2024-03-18T14:42:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.127+0000 7f95daffd640 1 -- 172.21.15.6:0/4161497986 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc035040 con 0x7f95e40d0450 2024-03-18T14:42:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.127+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e40cba30 con 0x7f95e40d0450 2024-03-18T14:42:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.128+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e40b6a40 con 0x7f95e40d0450 2024-03-18T14:42:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.128+0000 7f95daffd640 1 -- 172.21.15.6:0/4161497986 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f95cc038020 con 0x7f95e40d0450 2024-03-18T14:42:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.129+0000 7f95daffd640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f95c406bbd0 0x7f95c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.129+0000 7f95daffd640 1 -- 172.21.15.6:0/4161497986 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f95cc0ac170 con 0x7f95e40d0450 2024-03-18T14:42:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.129+0000 7f95e259c640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f95c406bbd0 0x7f95c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.129+0000 7f95e259c640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f95c406bbd0 0x7f95c406e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009f30 tx=0x7f95dc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.133+0000 7f95daffd640 1 -- 172.21.15.6:0/4161497986 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f95e40b6a40 con 0x7f95e40d0450 2024-03-18T14:42:05.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.316+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f95e40751b0 con 0x7f95e40d0450 2024-03-18T14:42:05.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.316+0000 7f95daffd640 1 -- 172.21.15.6:0/4161497986 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f95cc07a2d0 con 0x7f95e40d0450 2024-03-18T14:42:05.319 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f95c406bbd0 msgr2=0x7f95c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f95c406bbd0 0x7f95c406e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009f30 tx=0x7f95dc009190 comp rx=0 tx=0).stop 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e40d0450 msgr2=0x7f95e4159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e40d0450 0x7f95e4159380 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0098c0 tx=0x7f95cc0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 shutdown_connections 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f95c406bbd0 0x7f95c406e090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95e415e030 0x7f95e41586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95e40d0450 0x7f95e4159380 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 --2- 172.21.15.6:0/4161497986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95e40b6140 0x7f95e40cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 >> 172.21.15.6:0/4161497986 conn(0x7f95e406c8c0 msgr2=0x7f95e40d43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 shutdown_connections 2024-03-18T14:42:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.318+0000 7f95e8f48640 1 -- 172.21.15.6:0/4161497986 wait complete. 2024-03-18T14:42:05.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.470+0000 7f7c2ae27640 1 Processor -- start 2024-03-18T14:42:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.470+0000 7f7c2ae27640 1 -- start start 2024-03-18T14:42:05.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c2ae27640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c240b9650 0x7f7c240d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c2ae27640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c240b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c2ae27640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c240d0ff0 con 0x7f7c2415e170 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c2ae27640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7c240b9ae0 con 0x7f7c240b9650 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c2ae27640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240cef50 con 0x7f7c240d0bc0 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c23fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c240b9650 0x7f7c240d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c28bc3640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7c2415e170 0x7f7c240d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:59098/0 (socket says 172.21.15.6:59098) 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c28bc3640 1 -- 172.21.15.6:0/1824031752 learned_addr learned my addr 172.21.15.6:0/1824031752 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c293c4640 1 --2- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c240b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c23fff640 1 -- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 msgr2=0x7f7c240b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c23fff640 1 --2- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c240b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c23fff640 1 -- 172.21.15.6:0/1824031752 >> v1:172.21.15.92:6789/0 conn(0x7f7c2415e170 legacy=0x7f7c240d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c23fff640 1 -- 172.21.15.6:0/1824031752 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c240cf230 con 0x7f7c240b9650 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.471+0000 7f7c237fe640 1 -- 172.21.15.6:0/1824031752 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509371926 0 0) 0x7f7c240d0ff0 con 0x7f7c2415e170 2024-03-18T14:42:05.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c23fff640 1 --2- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c240b9650 0x7f7c240d0680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c100097a0 tx=0x7f7c1002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b96a783a144f3a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c237fe640 1 -- 172.21.15.6:0/1824031752 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10035020 con 0x7f7c240b9650 2024-03-18T14:42:05.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c237fe640 1 -- 172.21.15.6:0/1824031752 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c10031830 con 0x7f7c240b9650 2024-03-18T14:42:05.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c237fe640 1 -- 172.21.15.6:0/1824031752 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10031c00 con 0x7f7c240b9650 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c2ae27640 1 -- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c240b9650 msgr2=0x7f7c240d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c240b9650 0x7f7c240d0680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c100097a0 tx=0x7f7c1002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c2ae27640 1 -- 172.21.15.6:0/1824031752 shutdown_connections 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c240b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.472+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/1824031752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c240b9650 0x7f7c240d0680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.473+0000 7f7c2ae27640 1 -- 172.21.15.6:0/1824031752 >> 172.21.15.6:0/1824031752 conn(0x7f7c2406c900 msgr2=0x7f7c2406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.473+0000 7f7c2ae27640 1 -- 172.21.15.6:0/1824031752 shutdown_connections 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.473+0000 7f7c2ae27640 1 -- 172.21.15.6:0/1824031752 wait complete. 2024-03-18T14:42:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.474+0000 7f7c2ae27640 1 Processor -- start 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.474+0000 7f7c2ae27640 1 -- start start 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.474+0000 7f7c2ae27640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c240b9650 0x7f7c24158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.474+0000 7f7c2ae27640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c24155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.474+0000 7f7c2ae27640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2415e170 0x7f7c24156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.474+0000 7f7c2ae27640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240d3a40 con 0x7f7c240b9650 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.474+0000 7f7c2ae27640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240d2e10 con 0x7f7c2415e170 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c2ae27640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240d2820 con 0x7f7c240d0bc0 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c240b9650 0x7f7c24158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c240b9650 0x7f7c24158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39902/0 (socket says 172.21.15.6:39902) 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 -- 172.21.15.6:0/2952132229 learned_addr learned my addr 172.21.15.6:0/2952132229 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c23fff640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c24155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c293c4640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2415e170 0x7f7c24156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 -- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 msgr2=0x7f7c24155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c24155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 -- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2415e170 msgr2=0x7f7c24156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2415e170 0x7f7c24156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 -- 172.21.15.6:0/2952132229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c24158a80 con 0x7f7c240b9650 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c23fff640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c24155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c293c4640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2415e170 0x7f7c24156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c28bc3640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c240b9650 0x7f7c24158540 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c430 tx=0x7f7c1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c217fa640 1 -- 172.21.15.6:0/2952132229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c180074e0 con 0x7f7c240b9650 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c217fa640 1 -- 172.21.15.6:0/2952132229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c180047e0 con 0x7f7c240b9650 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c24156670 con 0x7f7c240b9650 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c217fa640 1 -- 172.21.15.6:0/2952132229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18004bd0 con 0x7f7c240b9650 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.475+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c24156c30 con 0x7f7c240b9650 2024-03-18T14:42:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.476+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2408afe0 con 0x7f7c240b9650 2024-03-18T14:42:05.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.477+0000 7f7c217fa640 1 -- 172.21.15.6:0/2952132229 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7c18011070 con 0x7f7c240b9650 2024-03-18T14:42:05.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.477+0000 7f7c217fa640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7c0806bbd0 0x7f7c0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.477+0000 7f7c217fa640 1 -- 172.21.15.6:0/2952132229 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7c1808a8d0 con 0x7f7c240b9650 2024-03-18T14:42:05.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.477+0000 7f7c23fff640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7c0806bbd0 0x7f7c0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.478+0000 7f7c23fff640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7c0806bbd0 0x7f7c0806e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c10036000 tx=0x7f7c10008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.481+0000 7f7c217fa640 1 -- 172.21.15.6:0/2952132229 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7c18058a30 con 0x7f7c240b9650 2024-03-18T14:42:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.704+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c2415d700 con 0x7f7c240b9650 2024-03-18T14:42:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.705+0000 7f7c217fa640 1 -- 172.21.15.6:0/2952132229 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7f7c1805cad0 con 0x7f7c240b9650 2024-03-18T14:42:05.708 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:05.708 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:42:05.708 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7c0806bbd0 msgr2=0x7f7c0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7c0806bbd0 0x7f7c0806e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c10036000 tx=0x7f7c10008030 comp rx=0 tx=0).stop 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c240b9650 msgr2=0x7f7c24158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c240b9650 0x7f7c24158540 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c430 tx=0x7f7c1800c8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 shutdown_connections 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7c0806bbd0 0x7f7c0806e090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c2415e170 0x7f7c24156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c240d0bc0 0x7f7c24155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 --2- 172.21.15.6:0/2952132229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c240b9650 0x7f7c24158540 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 >> 172.21.15.6:0/2952132229 conn(0x7f7c2406c900 msgr2=0x7f7c240b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 shutdown_connections 2024-03-18T14:42:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.707+0000 7f7c2ae27640 1 -- 172.21.15.6:0/2952132229 wait complete. 2024-03-18T14:42:05.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:05.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.819+0000 7f83b4e20640 1 Processor -- start 2024-03-18T14:42:05.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.819+0000 7f83b4e20640 1 -- start start 2024-03-18T14:42:05.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83b4e20640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b015e020 0x7f83b0162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83b4e20640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f83b00d3210 0x7f83b00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83b4e20640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b00b9660 con 0x7f83b00ced00 2024-03-18T14:42:05.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83b4e20640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f83b00d2270 con 0x7f83b00d3210 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83b4e20640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f83b00ce390 con 0x7f83b015e020 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83aed9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f83b00d3210 0x7f83b00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83add9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f83b00ced00 0x7f83b00d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38576/0 (socket says 172.21.15.6:38576) 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83add9b640 1 -- 172.21.15.6:0/302575532 learned_addr learned my addr 172.21.15.6:0/302575532 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83aed9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f83b00d3210 0x7f83b00d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39914/0 (socket says 172.21.15.6:39914) 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.820+0000 7f83ae59c640 1 --2- 172.21.15.6:0/302575532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b015e020 0x7f83b0162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.821+0000 7f83aed9d640 1 -- 172.21.15.6:0/302575532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b015e020 msgr2=0x7f83b0162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.821+0000 7f83aed9d640 1 --2- 172.21.15.6:0/302575532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b015e020 0x7f83b0162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.821+0000 7f83aed9d640 1 -- 172.21.15.6:0/302575532 >> v1:172.21.15.6:6789/0 conn(0x7f83b00ced00 legacy=0x7f83b00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.821+0000 7f83aed9d640 1 -- 172.21.15.6:0/302575532 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b00b9900 con 0x7f83b00d3210 2024-03-18T14:42:05.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.821+0000 7f83ad59a640 1 -- 172.21.15.6:0/302575532 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 829685081 0 0) 0x7f83b00b9660 con 0x7f83b00ced00 2024-03-18T14:42:05.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.821+0000 7f83aed9d640 1 --2- 172.21.15.6:0/302575532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b00d3210 0x7f83b00d3c30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f83b00d23f0 tx=0x7f83a002be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8c52bb24e8fc55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.821+0000 7f83ad59a640 1 -- 172.21.15.6:0/302575532 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0039070 con 0x7f83b00d3210 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83ad59a640 1 -- 172.21.15.6:0/302575532 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a002cbb0 con 0x7f83b00d3210 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83ad59a640 1 -- 172.21.15.6:0/302575532 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0034440 con 0x7f83b00d3210 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 -- 172.21.15.6:0/302575532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b00d3210 msgr2=0x7f83b00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 --2- 172.21.15.6:0/302575532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b00d3210 0x7f83b00d3c30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f83b00d23f0 tx=0x7f83a002be00 comp rx=0 tx=0).stop 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 -- 172.21.15.6:0/302575532 shutdown_connections 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 --2- 172.21.15.6:0/302575532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b00d3210 0x7f83b00d3c30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 --2- 172.21.15.6:0/302575532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b015e020 0x7f83b0162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 -- 172.21.15.6:0/302575532 >> 172.21.15.6:0/302575532 conn(0x7f83b0072ce0 msgr2=0x7f83b00730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 -- 172.21.15.6:0/302575532 shutdown_connections 2024-03-18T14:42:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.822+0000 7f83b4e20640 1 -- 172.21.15.6:0/302575532 wait complete. 2024-03-18T14:42:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.823+0000 7f83b4e20640 1 Processor -- start 2024-03-18T14:42:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.823+0000 7f83b4e20640 1 -- start start 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83b4e20640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b00ced00 0x7f83b0151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83b4e20640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b00d3210 0x7f83b00cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83b4e20640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b015e020 0x7f83b00ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83b4e20640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f83b00cf2c0 con 0x7f83b015e020 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83b4e20640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f83b00ce890 con 0x7f83b00d3210 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83b4e20640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f83b00cf440 con 0x7f83b00ced00 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83aed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b015e020 0x7f83b00ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83aed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b015e020 0x7f83b00ccc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39920/0 (socket says 172.21.15.6:39920) 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83aed9d640 1 -- 172.21.15.6:0/3317316116 learned_addr learned my addr 172.21.15.6:0/3317316116 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83add9b640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b00d3210 0x7f83b00cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.824+0000 7f83ae59c640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b00ced00 0x7f83b0151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83aed9d640 1 -- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b00ced00 msgr2=0x7f83b0151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83aed9d640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b00ced00 0x7f83b0151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83aed9d640 1 -- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b00d3210 msgr2=0x7f83b00cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83aed9d640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b00d3210 0x7f83b00cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83aed9d640 1 -- 172.21.15.6:0/3317316116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b01529d0 con 0x7f83b015e020 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83add9b640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b00d3210 0x7f83b00cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83aed9d640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b015e020 0x7f83b00ccc10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009e80 tx=0x7f83a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83a6ffd640 1 -- 172.21.15.6:0/3317316116 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0039050 con 0x7f83b015e020 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83a6ffd640 1 -- 172.21.15.6:0/3317316116 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a0045070 con 0x7f83b015e020 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83a6ffd640 1 -- 172.21.15.6:0/3317316116 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0009450 con 0x7f83b015e020 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b00b41e0 con 0x7f83b015e020 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.825+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b00b47a0 con 0x7f83b015e020 2024-03-18T14:42:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.826+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b008afe0 con 0x7f83b015e020 2024-03-18T14:42:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.826+0000 7f83a6ffd640 1 -- 172.21.15.6:0/3317316116 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f83a003e850 con 0x7f83b015e020 2024-03-18T14:42:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.827+0000 7f83a6ffd640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f839006bbd0 0x7f839006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.827+0000 7f83a6ffd640 1 -- 172.21.15.6:0/3317316116 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f83a00ada40 con 0x7f83b015e020 2024-03-18T14:42:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.827+0000 7f83ae59c640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f839006bbd0 0x7f839006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.827+0000 7f83ae59c640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f839006bbd0 0x7f839006e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009aa0 tx=0x7f83a8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:05.831+0000 7f83a6ffd640 1 -- 172.21.15.6:0/3317316116 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f83b01529d0 con 0x7f83b015e020 2024-03-18T14:42:06.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.012+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83b0068490 con 0x7f83b015e020 2024-03-18T14:42:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.014+0000 7f83a6ffd640 1 -- 172.21.15.6:0/3317316116 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7f83a007bcc0 con 0x7f83b015e020 2024-03-18T14:42:06.017 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:06.017 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f839006bbd0 msgr2=0x7f839006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f839006bbd0 0x7f839006e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009aa0 tx=0x7f83a8009190 comp rx=0 tx=0).stop 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b015e020 msgr2=0x7f83b00ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b015e020 0x7f83b00ccc10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009e80 tx=0x7f83a0004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 shutdown_connections 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f839006bbd0 0x7f839006e090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83b015e020 0x7f83b00ccc10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b00d3210 0x7f83b00cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 --2- 172.21.15.6:0/3317316116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b00ced00 0x7f83b0151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 >> 172.21.15.6:0/3317316116 conn(0x7f83b0072ce0 msgr2=0x7f83b0161b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 shutdown_connections 2024-03-18T14:42:06.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.016+0000 7f83b4e20640 1 -- 172.21.15.6:0/3317316116 wait complete. 2024-03-18T14:42:06.029 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:06.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.131+0000 7f22a1b1b640 1 Processor -- start 2024-03-18T14:42:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.132+0000 7f22a1b1b640 1 -- start start 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.132+0000 7f22a1b1b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f229c15dfd0 0x7f229c0b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.132+0000 7f22a1b1b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c0b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.132+0000 7f22a1b1b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229c15e3b0 con 0x7f229c0b5740 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.132+0000 7f22a1b1b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f229c0b8470 con 0x7f229c15dfd0 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.132+0000 7f22a1b1b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f229c0b8750 con 0x7f229c0b7e00 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.132+0000 7f229affd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f229c0b5740 0x7f229c0d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:59104/0 (socket says 172.21.15.6:59104) 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229affd640 1 -- 172.21.15.6:0/336634310 learned_addr learned my addr 172.21.15.6:0/336634310 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229b7fe640 1 --2- 172.21.15.6:0/336634310 >> v2:172.21.15.92:3300/0 conn(0x7f229c15dfd0 0x7f229c0b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229bfff640 1 --2- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c0b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229a7fc640 1 -- 172.21.15.6:0/336634310 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566634298 0 0) 0x7f229c15e3b0 con 0x7f229c0b5740 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229a7fc640 1 -- 172.21.15.6:0/336634310 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2284003520 con 0x7f229c0b5740 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229b7fe640 1 -- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 msgr2=0x7f229c0b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229b7fe640 1 --2- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c0b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229b7fe640 1 -- 172.21.15.6:0/336634310 >> v1:172.21.15.92:6789/0 conn(0x7f229c0b5740 legacy=0x7f229c0d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229b7fe640 1 -- 172.21.15.6:0/336634310 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229c0b8a30 con 0x7f229c15dfd0 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.133+0000 7f229bfff640 1 --2- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c0b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f229b7fe640 1 --2- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c15dfd0 0x7f229c0b9650 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f228c0097a0 tx=0x7f228c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f755dc0872371b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f229a7fc640 1 -- 172.21.15.6:0/336634310 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c035020 con 0x7f229c15dfd0 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f229a7fc640 1 -- 172.21.15.6:0/336634310 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228c031830 con 0x7f229c15dfd0 2024-03-18T14:42:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f229a7fc640 1 -- 172.21.15.6:0/336634310 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c031be0 con 0x7f229c15dfd0 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 -- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c15dfd0 msgr2=0x7f229c0b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c15dfd0 0x7f229c0b9650 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f228c0097a0 tx=0x7f228c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 -- 172.21.15.6:0/336634310 shutdown_connections 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c0b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/336634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c15dfd0 0x7f229c0b9650 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 -- 172.21.15.6:0/336634310 >> 172.21.15.6:0/336634310 conn(0x7f229c072d10 msgr2=0x7f229c073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 -- 172.21.15.6:0/336634310 shutdown_connections 2024-03-18T14:42:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.134+0000 7f22a1b1b640 1 -- 172.21.15.6:0/336634310 wait complete. 2024-03-18T14:42:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.135+0000 7f22a1b1b640 1 Processor -- start 2024-03-18T14:42:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f22a1b1b640 1 -- start start 2024-03-18T14:42:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f22a1b1b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c0b5740 0x7f229c15af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f22a1b1b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f22a1b1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f229c15dfd0 0x7f229c15c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f22a1b1b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f229c0d2550 con 0x7f229c15dfd0 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f22a1b1b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f229c0b9b90 con 0x7f229c0b5740 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f22a1b1b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f229c0d26d0 con 0x7f229c0b7e00 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f229bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f229c15dfd0 0x7f229c15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f229affd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f229b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c0b5740 0x7f229c15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f229bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f229c15dfd0 0x7f229c15c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39930/0 (socket says 172.21.15.6:39930) 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.136+0000 7f229bfff640 1 -- 172.21.15.6:0/959030789 learned_addr learned my addr 172.21.15.6:0/959030789 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f229b7fe640 1 -- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 msgr2=0x7f229c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f229b7fe640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f229b7fe640 1 -- 172.21.15.6:0/959030789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f229c15dfd0 msgr2=0x7f229c15c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f229b7fe640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f229c15dfd0 0x7f229c15c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f229b7fe640 1 -- 172.21.15.6:0/959030789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229c15b450 con 0x7f229c0b5740 2024-03-18T14:42:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f229bfff640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f229c15dfd0 0x7f229c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f229b7fe640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c0b5740 0x7f229c15af10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f228c033000 tx=0x7f228c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f2283fff640 1 -- 172.21.15.6:0/959030789 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c040070 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f2283fff640 1 -- 172.21.15.6:0/959030789 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228c03a830 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229c15f960 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f2283fff640 1 -- 172.21.15.6:0/959030789 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c035040 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.137+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229c15ff20 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.138+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f229c0751b0 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.138+0000 7f2283fff640 1 -- 172.21.15.6:0/959030789 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f228c038020 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.139+0000 7f2283fff640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f228806bbd0 0x7f228806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.139+0000 7f2283fff640 1 -- 172.21.15.6:0/959030789 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f228c0ac3d0 con 0x7f229c0b5740 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.139+0000 7f229affd640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f228806bbd0 0x7f228806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.139+0000 7f229affd640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f228806bbd0 0x7f228806e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f229c0c75e0 tx=0x7f2290009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.148+0000 7f2283fff640 1 -- 172.21.15.6:0/959030789 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f228c07a530 con 0x7f229c0b5740 2024-03-18T14:42:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.330+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f229c1605e0 con 0x7f229c0b5740 2024-03-18T14:42:06.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.331+0000 7f2283fff640 1 -- 172.21.15.6:0/959030789 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f228c07e5d0 con 0x7f229c0b5740 2024-03-18T14:42:06.334 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f228806bbd0 msgr2=0x7f228806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f228806bbd0 0x7f228806e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f229c0c75e0 tx=0x7f2290009040 comp rx=0 tx=0).stop 2024-03-18T14:42:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c0b5740 msgr2=0x7f229c15af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c0b5740 0x7f229c15af10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f228c033000 tx=0x7f228c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 shutdown_connections 2024-03-18T14:42:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f228806bbd0 0x7f228806e090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f229c15dfd0 0x7f229c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229c0b7e00 0x7f229c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 --2- 172.21.15.6:0/959030789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f229c0b5740 0x7f229c15af10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 >> 172.21.15.6:0/959030789 conn(0x7f229c072d10 msgr2=0x7f229c0d39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 shutdown_connections 2024-03-18T14:42:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.333+0000 7f22a1b1b640 1 -- 172.21.15.6:0/959030789 wait complete. 2024-03-18T14:42:06.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.449+0000 7ff9c8f87640 1 Processor -- start 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.449+0000 7ff9c8f87640 1 -- start start 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c8f87640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff9c415dfd0 0x7ff9c40b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c8f87640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c415e3b0 con 0x7ff9c40b7e00 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c8f87640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c40b8470 con 0x7ff9c40b5740 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c8f87640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c40b8240 con 0x7ff9c415dfd0 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c2d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff9c40b7e00 0x7ff9c40d2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38582/0 (socket says 172.21.15.6:38582) 2024-03-18T14:42:06.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c2d9d640 1 -- 172.21.15.6:0/1416305759 learned_addr learned my addr 172.21.15.6:0/1416305759 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:06.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c259c640 1 --2- 172.21.15.6:0/1416305759 >> v2:172.21.15.92:3300/0 conn(0x7ff9c415dfd0 0x7ff9c40b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c1d9b640 1 -- 172.21.15.6:0/1416305759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1949600652 0 0) 0x7ff9c415e3b0 con 0x7ff9c40b7e00 2024-03-18T14:42:06.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c1d9b640 1 -- 172.21.15.6:0/1416305759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9b0003520 con 0x7ff9c40b7e00 2024-03-18T14:42:06.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c1d9b640 1 -- 172.21.15.6:0/1416305759 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2844571799 0 0) 0x7ff9c40b8470 con 0x7ff9c40b5740 2024-03-18T14:42:06.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c259c640 1 -- 172.21.15.6:0/1416305759 >> v1:172.21.15.92:6789/0 conn(0x7ff9c40b5740 legacy=0x7ff9c40d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c259c640 1 -- 172.21.15.6:0/1416305759 >> v1:172.21.15.6:6789/0 conn(0x7ff9c40b7e00 legacy=0x7ff9c40d2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.450+0000 7ff9c259c640 1 -- 172.21.15.6:0/1416305759 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c40b8750 con 0x7ff9c415dfd0 2024-03-18T14:42:06.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.451+0000 7ff9c259c640 1 --2- 172.21.15.6:0/1416305759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c40b9650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40097a0 tx=0x7ff9b402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=493a98e4a72e7143 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.451+0000 7ff9c1d9b640 1 -- 172.21.15.6:0/1416305759 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4034070 con 0x7ff9c415dfd0 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.451+0000 7ff9c1d9b640 1 -- 172.21.15.6:0/1416305759 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b4030610 con 0x7ff9c415dfd0 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.451+0000 7ff9c1d9b640 1 -- 172.21.15.6:0/1416305759 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40309e0 con 0x7ff9c415dfd0 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.452+0000 7ff9c8f87640 1 -- 172.21.15.6:0/1416305759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 msgr2=0x7ff9c40b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.452+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/1416305759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c40b9650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40097a0 tx=0x7ff9b402b960 comp rx=0 tx=0).stop 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.452+0000 7ff9c8f87640 1 -- 172.21.15.6:0/1416305759 shutdown_connections 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.452+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/1416305759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c40b9650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.452+0000 7ff9c8f87640 1 -- 172.21.15.6:0/1416305759 >> 172.21.15.6:0/1416305759 conn(0x7ff9c4072d10 msgr2=0x7ff9c4073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.452+0000 7ff9c8f87640 1 -- 172.21.15.6:0/1416305759 shutdown_connections 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.452+0000 7ff9c8f87640 1 -- 172.21.15.6:0/1416305759 wait complete. 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.453+0000 7ff9c8f87640 1 Processor -- start 2024-03-18T14:42:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.453+0000 7ff9c8f87640 1 -- start start 2024-03-18T14:42:06.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c8f87640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9c40b5740 0x7ff9c4153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c8f87640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9c40b7e00 0x7ff9c40c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c8f87640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c41f7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c8f87640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c40b9b90 con 0x7ff9c40b7e00 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c8f87640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c40b8d10 con 0x7ff9c415dfd0 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c8f87640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c40b5bd0 con 0x7ff9c40b5740 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9c40b5740 0x7ff9c4153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c2d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c41f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9bbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9c40b7e00 0x7ff9c40c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9c40b5740 0x7ff9c4153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40458/0 (socket says 172.21.15.6:40458) 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 -- 172.21.15.6:0/2194290747 learned_addr learned my addr 172.21.15.6:0/2194290747 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 -- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 msgr2=0x7ff9c41f7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c41f7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 -- 172.21.15.6:0/2194290747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9c40b7e00 msgr2=0x7ff9c40c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9c40b7e00 0x7ff9c40c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 -- 172.21.15.6:0/2194290747 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c4155ff0 con 0x7ff9c40b5740 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9bbfff640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9c40b7e00 0x7ff9c40c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c2d9d640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c41f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.454+0000 7ff9c259c640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9c40b5740 0x7ff9c4153970 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4000c00 tx=0x7ff9b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.455+0000 7ff9baffd640 1 -- 172.21.15.6:0/2194290747 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4034040 con 0x7ff9c40b5740 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.455+0000 7ff9baffd640 1 -- 172.21.15.6:0/2194290747 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b403aaa0 con 0x7ff9c40b5740 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.455+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c41562d0 con 0x7ff9c40b5740 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.455+0000 7ff9baffd640 1 -- 172.21.15.6:0/2194290747 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40443f0 con 0x7ff9c40b5740 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.455+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c4156890 con 0x7ff9c40b5740 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.456+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c40751b0 con 0x7ff9c40b5740 2024-03-18T14:42:06.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.456+0000 7ff9baffd640 1 -- 172.21.15.6:0/2194290747 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff9b404b060 con 0x7ff9c40b5740 2024-03-18T14:42:06.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.456+0000 7ff9baffd640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9a406bbd0 0x7ff9a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.457+0000 7ff9baffd640 1 -- 172.21.15.6:0/2194290747 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff9b40ac9f0 con 0x7ff9c40b5740 2024-03-18T14:42:06.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.457+0000 7ff9bbfff640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9a406bbd0 0x7ff9a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.457+0000 7ff9bbfff640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9a406bbd0 0x7ff9a406e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff9c415d4a0 tx=0x7ff9ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.461+0000 7ff9baffd640 1 -- 172.21.15.6:0/2194290747 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff9b403f070 con 0x7ff9c40b5740 2024-03-18T14:42:06.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.644+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff9c40cb170 con 0x7ff9c40b5740 2024-03-18T14:42:06.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.644+0000 7ff9baffd640 1 -- 172.21.15.6:0/2194290747 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7ff9b407ab60 con 0x7ff9c40b5740 2024-03-18T14:42:06.647 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.646+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9a406bbd0 msgr2=0x7ff9a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.646+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9a406bbd0 0x7ff9a406e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff9c415d4a0 tx=0x7ff9ac009040 comp rx=0 tx=0).stop 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.646+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9c40b5740 msgr2=0x7ff9c4153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.646+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9c40b5740 0x7ff9c4153970 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4000c00 tx=0x7ff9b40046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 shutdown_connections 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff9a406bbd0 0x7ff9a406e090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9c415dfd0 0x7ff9c41f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9c40b7e00 0x7ff9c40c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 --2- 172.21.15.6:0/2194290747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9c40b5740 0x7ff9c4153970 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 >> 172.21.15.6:0/2194290747 conn(0x7ff9c4072d10 msgr2=0x7ff9c40c7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 shutdown_connections 2024-03-18T14:42:06.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.647+0000 7ff9c8f87640 1 -- 172.21.15.6:0/2194290747 wait complete. 2024-03-18T14:42:06.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.770+0000 7fb9b8ec6640 1 Processor -- start 2024-03-18T14:42:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.770+0000 7fb9b8ec6640 1 -- start start 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b8ec6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9b415e030 0x7fb9b40d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b8ec6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40d0450 0x7fb9b40d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b8ec6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b40d0e20 con 0x7fb9b40b6140 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b8ec6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb9b415e410 con 0x7fb9b415e030 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b8ec6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b40ce300 con 0x7fb9b40d0450 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b259c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9b415e030 0x7fb9b40d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b1d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb9b40b6140 0x7fb9b40ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38596/0 (socket says 172.21.15.6:38596) 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b1d9b640 1 -- 172.21.15.6:0/3557457031 learned_addr learned my addr 172.21.15.6:0/3557457031 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b2d9d640 1 --2- 172.21.15.6:0/3557457031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40d0450 0x7fb9b40d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b259c640 1 -- 172.21.15.6:0/3557457031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40d0450 msgr2=0x7fb9b40d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b259c640 1 --2- 172.21.15.6:0/3557457031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40d0450 0x7fb9b40d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b259c640 1 -- 172.21.15.6:0/3557457031 >> v1:172.21.15.6:6789/0 conn(0x7fb9b40b6140 legacy=0x7fb9b40ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b159a640 1 -- 172.21.15.6:0/3557457031 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 689843800 0 0) 0x7fb9b40d0e20 con 0x7fb9b40b6140 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.771+0000 7fb9b259c640 1 -- 172.21.15.6:0/3557457031 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b40b65a0 con 0x7fb9b415e030 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.772+0000 7fb9b259c640 1 --2- 172.21.15.6:0/3557457031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40d3570 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb9a40097a0 tx=0x7fb9a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af24a07a5eb7d3c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.772+0000 7fb9b159a640 1 -- 172.21.15.6:0/3557457031 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4035020 con 0x7fb9b415e030 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.772+0000 7fb9b159a640 1 -- 172.21.15.6:0/3557457031 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a4031830 con 0x7fb9b415e030 2024-03-18T14:42:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.772+0000 7fb9b159a640 1 -- 172.21.15.6:0/3557457031 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a4031c00 con 0x7fb9b415e030 2024-03-18T14:42:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/3557457031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 msgr2=0x7fb9b40d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/3557457031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40d3570 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb9a40097a0 tx=0x7fb9a402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/3557457031 shutdown_connections 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/3557457031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40d0450 0x7fb9b40d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/3557457031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40d3570 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/3557457031 >> 172.21.15.6:0/3557457031 conn(0x7fb9b406c8c0 msgr2=0x7fb9b406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/3557457031 shutdown_connections 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.773+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/3557457031 wait complete. 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 Processor -- start 2024-03-18T14:42:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 -- start start 2024-03-18T14:42:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40b6140 0x7fb9b415cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9b40d0450 0x7fb9b4160be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b40ce550 con 0x7fb9b415e030 2024-03-18T14:42:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b40d3290 con 0x7fb9b40d0450 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.775+0000 7fb9b8ec6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b40ce830 con 0x7fb9b40b6140 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39950/0 (socket says 172.21.15.6:39950) 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 -- 172.21.15.6:0/78334709 learned_addr learned my addr 172.21.15.6:0/78334709 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b1d9b640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9b40d0450 0x7fb9b4160be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b259c640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40b6140 0x7fb9b415cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 -- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40b6140 msgr2=0x7fb9b415cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40b6140 0x7fb9b415cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 -- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9b40d0450 msgr2=0x7fb9b4160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9b40d0450 0x7fb9b4160be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 -- 172.21.15.6:0/78334709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b4161550 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b1d9b640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9b40d0450 0x7fb9b4160be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b2d9d640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40cbdd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009dd0 tx=0x7fb9ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9aaffd640 1 -- 172.21.15.6:0/78334709 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac018020 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9aaffd640 1 -- 172.21.15.6:0/78334709 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac014480 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b41e7a90 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.776+0000 7fb9aaffd640 1 -- 172.21.15.6:0/78334709 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac014870 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.777+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b41e8050 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.778+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b40751b0 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.778+0000 7fb9aaffd640 1 -- 172.21.15.6:0/78334709 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb9ac01b020 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.778+0000 7fb9aaffd640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb99406bbd0 0x7fb99406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.779+0000 7fb9aaffd640 1 -- 172.21.15.6:0/78334709 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb9ac08b790 con 0x7fb9b415e030 2024-03-18T14:42:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.779+0000 7fb9b259c640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb99406bbd0 0x7fb99406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.779+0000 7fb9b259c640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb99406bbd0 0x7fb99406e090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4036000 tx=0x7fb9a403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.783+0000 7fb9aaffd640 1 -- 172.21.15.6:0/78334709 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb9ac022030 con 0x7fb9b415e030 2024-03-18T14:42:06.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.965+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb9b41e8220 con 0x7fb9b415e030 2024-03-18T14:42:06.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.966+0000 7fb9aaffd640 1 -- 172.21.15.6:0/78334709 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fb9ac010040 con 0x7fb9b415e030 2024-03-18T14:42:06.968 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb99406bbd0 msgr2=0x7fb99406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb99406bbd0 0x7fb99406e090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4036000 tx=0x7fb9a403d040 comp rx=0 tx=0).stop 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 msgr2=0x7fb9b40cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40cbdd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009dd0 tx=0x7fb9ac00c490 comp rx=0 tx=0).stop 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 shutdown_connections 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb99406bbd0 0x7fb99406e090 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b415e030 0x7fb9b40cbdd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9b40d0450 0x7fb9b4160be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 --2- 172.21.15.6:0/78334709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9b40b6140 0x7fb9b415cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.967+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 >> 172.21.15.6:0/78334709 conn(0x7fb9b406c8c0 msgr2=0x7fb9b40b7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.968+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 shutdown_connections 2024-03-18T14:42:06.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:06.968+0000 7fb9b8ec6640 1 -- 172.21.15.6:0/78334709 wait complete. 2024-03-18T14:42:06.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.083+0000 7fafd8aff640 1 Processor -- start 2024-03-18T14:42:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.083+0000 7fafd8aff640 1 -- start start 2024-03-18T14:42:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd8aff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fafd415e030 0x7fafd40d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd8aff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd40b6140 0x7fafd40ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd8aff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fafd40d0450 0x7fafd40d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd8aff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fafd415e410 con 0x7fafd415e030 2024-03-18T14:42:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd8aff640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fafd40ce300 con 0x7fafd40d0450 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd8aff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fafd40d0e20 con 0x7fafd40b6140 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd259c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fafd415e030 0x7fafd40d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd2d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fafd40d0450 0x7fafd40d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd259c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fafd415e030 0x7fafd40d3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39964/0 (socket says 172.21.15.6:39964) 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd259c640 1 -- 172.21.15.6:0/1048973245 learned_addr learned my addr 172.21.15.6:0/1048973245 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.084+0000 7fafd1d9b640 1 --2- 172.21.15.6:0/1048973245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd40b6140 0x7fafd40ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd259c640 1 -- 172.21.15.6:0/1048973245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd40b6140 msgr2=0x7fafd40ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd259c640 1 --2- 172.21.15.6:0/1048973245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd40b6140 0x7fafd40ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd259c640 1 -- 172.21.15.6:0/1048973245 >> v2:172.21.15.92:3300/0 conn(0x7fafd40d0450 msgr2=0x7fafd40d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd259c640 1 --2- 172.21.15.6:0/1048973245 >> v2:172.21.15.92:3300/0 conn(0x7fafd40d0450 0x7fafd40d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd259c640 1 -- 172.21.15.6:0/1048973245 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd40b65a0 con 0x7fafd415e030 2024-03-18T14:42:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd2d9d640 1 --2- 172.21.15.6:0/1048973245 >> v2:172.21.15.92:3300/0 conn(0x7fafd40d0450 0x7fafd40d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd1d9b640 1 --2- 172.21.15.6:0/1048973245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd40b6140 0x7fafd40ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.085+0000 7fafd259c640 1 --2- 172.21.15.6:0/1048973245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd415e030 0x7fafd40d3570 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fafd40b9730 tx=0x7fafcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e0909b9eeb1d87 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd159a640 1 -- 172.21.15.6:0/1048973245 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc035020 con 0x7fafd415e030 2024-03-18T14:42:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd159a640 1 -- 172.21.15.6:0/1048973245 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafcc031870 con 0x7fafd415e030 2024-03-18T14:42:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd159a640 1 -- 172.21.15.6:0/1048973245 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc031c40 con 0x7fafd415e030 2024-03-18T14:42:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 -- 172.21.15.6:0/1048973245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd415e030 msgr2=0x7fafd40d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 --2- 172.21.15.6:0/1048973245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd415e030 0x7fafd40d3570 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fafd40b9730 tx=0x7fafcc02b910 comp rx=0 tx=0).stop 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 -- 172.21.15.6:0/1048973245 shutdown_connections 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 --2- 172.21.15.6:0/1048973245 >> v2:172.21.15.92:3300/0 conn(0x7fafd40d0450 0x7fafd40d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 --2- 172.21.15.6:0/1048973245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd40b6140 0x7fafd40ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 --2- 172.21.15.6:0/1048973245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd415e030 0x7fafd40d3570 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 -- 172.21.15.6:0/1048973245 >> 172.21.15.6:0/1048973245 conn(0x7fafd406c8c0 msgr2=0x7fafd406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 -- 172.21.15.6:0/1048973245 shutdown_connections 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.086+0000 7fafd8aff640 1 -- 172.21.15.6:0/1048973245 wait complete. 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 Processor -- start 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 -- start start 2024-03-18T14:42:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd40b6140 0x7fafd40cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd40d0450 0x7fafd40cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd415e030 0x7fafd40cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fafd40ce550 con 0x7fafd40d0450 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fafd40d3290 con 0x7fafd40b6140 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd8aff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fafd40ce830 con 0x7fafd415e030 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd1d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd40d0450 0x7fafd40cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd1d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd40d0450 0x7fafd40cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39966/0 (socket says 172.21.15.6:39966) 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd1d9b640 1 -- 172.21.15.6:0/4212818805 learned_addr learned my addr 172.21.15.6:0/4212818805 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.088+0000 7fafd259c640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd40b6140 0x7fafd40cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd2d9d640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd415e030 0x7fafd40cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd1d9b640 1 -- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd415e030 msgr2=0x7fafd40cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd1d9b640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd415e030 0x7fafd40cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd1d9b640 1 -- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd40b6140 msgr2=0x7fafd40cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd1d9b640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd40b6140 0x7fafd40cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd1d9b640 1 -- 172.21.15.6:0/4212818805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd40ccbf0 con 0x7fafd40d0450 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd1d9b640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd40d0450 0x7fafd40cad80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00e980 tx=0x7fafbc00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafcaffd640 1 -- 172.21.15.6:0/4212818805 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc017070 con 0x7fafd40d0450 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafcaffd640 1 -- 172.21.15.6:0/4212818805 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafbc00c940 con 0x7fafd40d0450 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafcaffd640 1 -- 172.21.15.6:0/4212818805 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00cd10 con 0x7fafd40d0450 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd40cb610 con 0x7fafd40d0450 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.089+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd40cbeb0 con 0x7fafd40d0450 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.090+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd40751b0 con 0x7fafd40d0450 2024-03-18T14:42:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.091+0000 7fafcaffd640 1 -- 172.21.15.6:0/4212818805 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fafbc01a070 con 0x7fafd40d0450 2024-03-18T14:42:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.091+0000 7fafcaffd640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fafb406bbd0 0x7fafb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.091+0000 7fafcaffd640 1 -- 172.21.15.6:0/4212818805 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fafbc08aa80 con 0x7fafd40d0450 2024-03-18T14:42:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.091+0000 7fafd259c640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fafb406bbd0 0x7fafb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.092+0000 7fafd259c640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fafb406bbd0 0x7fafb406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fafcc036000 tx=0x7fafcc03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.095+0000 7fafcaffd640 1 -- 172.21.15.6:0/4212818805 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fafbc058c70 con 0x7fafd40d0450 2024-03-18T14:42:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.276+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fafd40b9920 con 0x7fafd40d0450 2024-03-18T14:42:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.277+0000 7fafcaffd640 1 -- 172.21.15.6:0/4212818805 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fafbc05cd10 con 0x7fafd40d0450 2024-03-18T14:42:07.280 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fafb406bbd0 msgr2=0x7fafb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fafb406bbd0 0x7fafb406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fafcc036000 tx=0x7fafcc03c040 comp rx=0 tx=0).stop 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd40d0450 msgr2=0x7fafd40cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd40d0450 0x7fafd40cad80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00e980 tx=0x7fafbc00ed80 comp rx=0 tx=0).stop 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 shutdown_connections 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fafb406bbd0 0x7fafb406e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fafd415e030 0x7fafd40cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafd40d0450 0x7fafd40cad80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 --2- 172.21.15.6:0/4212818805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd40b6140 0x7fafd40cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 >> 172.21.15.6:0/4212818805 conn(0x7fafd406c8c0 msgr2=0x7fafd4159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:07.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 shutdown_connections 2024-03-18T14:42:07.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.279+0000 7fafd8aff640 1 -- 172.21.15.6:0/4212818805 wait complete. 2024-03-18T14:42:07.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.394+0000 7f04e286c640 1 Processor -- start 2024-03-18T14:42:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.394+0000 7f04e286c640 1 -- start start 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.394+0000 7f04e286c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04dc15e020 0x7f04dc162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.394+0000 7f04e286c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f04dc0d3210 0x7f04dc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.394+0000 7f04e286c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04dc0b9660 con 0x7f04dc0ced00 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04e286c640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f04dc0d2270 con 0x7f04dc15e020 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04e286c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f04dc0ce390 con 0x7f04dc0d3210 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04dbfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04dc15e020 0x7f04dc162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04db7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f04dc0ced00 0x7f04dc0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38598/0 (socket says 172.21.15.6:38598) 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04dbfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04dc15e020 0x7f04dc162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39968/0 (socket says 172.21.15.6:39968) 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04db7fe640 1 -- 172.21.15.6:0/2202311656 learned_addr learned my addr 172.21.15.6:0/2202311656 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04e0e09640 1 --2- 172.21.15.6:0/2202311656 >> v2:172.21.15.92:3300/0 conn(0x7f04dc0d3210 0x7f04dc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04dbfff640 1 -- 172.21.15.6:0/2202311656 >> v2:172.21.15.92:3300/0 conn(0x7f04dc0d3210 msgr2=0x7f04dc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04dbfff640 1 --2- 172.21.15.6:0/2202311656 >> v2:172.21.15.92:3300/0 conn(0x7f04dc0d3210 0x7f04dc0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04dbfff640 1 -- 172.21.15.6:0/2202311656 >> v1:172.21.15.6:6789/0 conn(0x7f04dc0ced00 legacy=0x7f04dc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04dbfff640 1 -- 172.21.15.6:0/2202311656 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04dc0b9900 con 0x7f04dc15e020 2024-03-18T14:42:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.395+0000 7f04daffd640 1 -- 172.21.15.6:0/2202311656 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566797018 0 0) 0x7f04dc0b9660 con 0x7f04dc0ced00 2024-03-18T14:42:07.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04dbfff640 1 --2- 172.21.15.6:0/2202311656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc162380 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04dc15e400 tx=0x7f04d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c256f4fcac835e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04daffd640 1 -- 172.21.15.6:0/2202311656 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d00315a0 con 0x7f04dc15e020 2024-03-18T14:42:07.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04daffd640 1 -- 172.21.15.6:0/2202311656 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d0037070 con 0x7f04dc15e020 2024-03-18T14:42:07.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04daffd640 1 -- 172.21.15.6:0/2202311656 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0031960 con 0x7f04dc15e020 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 -- 172.21.15.6:0/2202311656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 msgr2=0x7f04dc162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 --2- 172.21.15.6:0/2202311656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc162380 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04dc15e400 tx=0x7f04d002b840 comp rx=0 tx=0).stop 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 -- 172.21.15.6:0/2202311656 shutdown_connections 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 --2- 172.21.15.6:0/2202311656 >> v2:172.21.15.92:3300/0 conn(0x7f04dc0d3210 0x7f04dc0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 --2- 172.21.15.6:0/2202311656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc162380 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 -- 172.21.15.6:0/2202311656 >> 172.21.15.6:0/2202311656 conn(0x7f04dc072ce0 msgr2=0x7f04dc0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 -- 172.21.15.6:0/2202311656 shutdown_connections 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.396+0000 7f04e286c640 1 -- 172.21.15.6:0/2202311656 wait complete. 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 Processor -- start 2024-03-18T14:42:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 -- start start 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04dc0ced00 0x7f04dc1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04dc0d3210 0x7f04dc0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc0cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc0cf2c0 con 0x7f04dc15e020 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc0ce890 con 0x7f04dc0d3210 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.398+0000 7f04e286c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f04dc0cf440 con 0x7f04dc0ced00 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc0cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc0cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39984/0 (socket says 172.21.15.6:39984) 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 -- 172.21.15.6:0/298373989 learned_addr learned my addr 172.21.15.6:0/298373989 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04dbfff640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04dc0ced00 0x7f04dc1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04db7fe640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04dc0d3210 0x7f04dc0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 -- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04dc0ced00 msgr2=0x7f04dc1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04dc0ced00 0x7f04dc1552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 -- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04dc0d3210 msgr2=0x7f04dc0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04dc0d3210 0x7f04dc0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 -- 172.21.15.6:0/298373989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04dc153540 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e0e09640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc0cd340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04d400efc0 tx=0x7f04d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04d8ff9640 1 -- 172.21.15.6:0/298373989 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4018020 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04d8ff9640 1 -- 172.21.15.6:0/298373989 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d4014480 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04dc153820 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04d8ff9640 1 -- 172.21.15.6:0/298373989 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4014870 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.399+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04dc153de0 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.400+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04dc08afe0 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.401+0000 7f04d8ff9640 1 -- 172.21.15.6:0/298373989 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f04d401b020 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.401+0000 7f04d8ff9640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f04c806bbd0 0x7f04c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.401+0000 7f04d8ff9640 1 -- 172.21.15.6:0/298373989 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f04d408b850 con 0x7f04dc15e020 2024-03-18T14:42:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.401+0000 7f04dbfff640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f04c806bbd0 0x7f04c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.402+0000 7f04dbfff640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f04c806bbd0 0x7f04c806e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f04d0036000 tx=0x7f04d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.405+0000 7f04d8ff9640 1 -- 172.21.15.6:0/298373989 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f04d4022030 con 0x7f04dc15e020 2024-03-18T14:42:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.627+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04dc0ca600 con 0x7f04dc15e020 2024-03-18T14:42:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.631+0000 7f04d8ff9640 1 -- 172.21.15.6:0/298373989 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7f04d4010040 con 0x7f04dc15e020 2024-03-18T14:42:07.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:07.634 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:42:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:42:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f04c806bbd0 msgr2=0x7f04c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f04c806bbd0 0x7f04c806e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f04d0036000 tx=0x7f04d0008030 comp rx=0 tx=0).stop 2024-03-18T14:42:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 msgr2=0x7f04dc0cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc0cd340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04d400efc0 tx=0x7f04d400c490 comp rx=0 tx=0).stop 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 shutdown_connections 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f04c806bbd0 0x7f04c806e090 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04dc15e020 0x7f04dc0cd340 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04dc0d3210 0x7f04dc0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 --2- 172.21.15.6:0/298373989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04dc0ced00 0x7f04dc1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.633+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 >> 172.21.15.6:0/298373989 conn(0x7f04dc072ce0 msgr2=0x7f04dc0d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.634+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 shutdown_connections 2024-03-18T14:42:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.634+0000 7f04e286c640 1 -- 172.21.15.6:0/298373989 wait complete. 2024-03-18T14:42:07.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:07.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.746+0000 7ff3eae44640 1 Processor -- start 2024-03-18T14:42:07.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.746+0000 7ff3eae44640 1 -- start start 2024-03-18T14:42:07.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3eae44640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3e415e170 0x7ff3e40b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3eae44640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff3e40d3330 0x7ff3e40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3eae44640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3e40cef50 con 0x7ff3e40d37c0 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3eae44640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff3e40b9b90 con 0x7ff3e415e170 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3eae44640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff3e40cf230 con 0x7ff3e40d3330 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e8be0640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3e415e170 0x7ff3e40b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e93e1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff3e40d37c0 0x7ff3e40b6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38610/0 (socket says 172.21.15.6:38610) 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e93e1640 1 -- 172.21.15.6:0/2439428407 learned_addr learned my addr 172.21.15.6:0/2439428407 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e3fff640 1 --2- 172.21.15.6:0/2439428407 >> v2:172.21.15.92:3300/0 conn(0x7ff3e40d3330 0x7ff3e40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e8be0640 1 -- 172.21.15.6:0/2439428407 >> v2:172.21.15.92:3300/0 conn(0x7ff3e40d3330 msgr2=0x7ff3e40d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e8be0640 1 --2- 172.21.15.6:0/2439428407 >> v2:172.21.15.92:3300/0 conn(0x7ff3e40d3330 0x7ff3e40d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e8be0640 1 -- 172.21.15.6:0/2439428407 >> v1:172.21.15.6:6789/0 conn(0x7ff3e40d37c0 legacy=0x7ff3e40b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e37fe640 1 -- 172.21.15.6:0/2439428407 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028191833 0 0) 0x7ff3e40cef50 con 0x7ff3e40d37c0 2024-03-18T14:42:07.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.747+0000 7ff3e8be0640 1 -- 172.21.15.6:0/2439428407 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e40d2b30 con 0x7ff3e415e170 2024-03-18T14:42:07.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.748+0000 7ff3e8be0640 1 --2- 172.21.15.6:0/2439428407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e415e170 0x7ff3e40b9650 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40d1740 tx=0x7ff3d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4713d94b33ef802f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.748+0000 7ff3e37fe640 1 -- 172.21.15.6:0/2439428407 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8035020 con 0x7ff3e415e170 2024-03-18T14:42:07.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.748+0000 7ff3e37fe640 1 -- 172.21.15.6:0/2439428407 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d8031830 con 0x7ff3e415e170 2024-03-18T14:42:07.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.748+0000 7ff3e37fe640 1 -- 172.21.15.6:0/2439428407 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8031c00 con 0x7ff3e415e170 2024-03-18T14:42:07.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 -- 172.21.15.6:0/2439428407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e415e170 msgr2=0x7ff3e40b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 --2- 172.21.15.6:0/2439428407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e415e170 0x7ff3e40b9650 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40d1740 tx=0x7ff3d802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 -- 172.21.15.6:0/2439428407 shutdown_connections 2024-03-18T14:42:07.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 --2- 172.21.15.6:0/2439428407 >> v2:172.21.15.92:3300/0 conn(0x7ff3e40d3330 0x7ff3e40d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 --2- 172.21.15.6:0/2439428407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e415e170 0x7ff3e40b9650 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 -- 172.21.15.6:0/2439428407 >> 172.21.15.6:0/2439428407 conn(0x7ff3e406c900 msgr2=0x7ff3e406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 -- 172.21.15.6:0/2439428407 shutdown_connections 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.749+0000 7ff3eae44640 1 -- 172.21.15.6:0/2439428407 wait complete. 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.750+0000 7ff3eae44640 1 Processor -- start 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.750+0000 7ff3eae44640 1 -- start start 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e40d3330 0x7ff3e40ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3e40d37c0 0x7ff3e414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3e415e170 0x7ff3e41573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e40ceb70 con 0x7ff3e40d3330 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e40d2820 con 0x7ff3e40d37c0 2024-03-18T14:42:07.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e40cecf0 con 0x7ff3e415e170 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e40d3330 0x7ff3e40ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e40d3330 0x7ff3e40ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39990/0 (socket says 172.21.15.6:39990) 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 -- 172.21.15.6:0/233800455 learned_addr learned my addr 172.21.15.6:0/233800455 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e3fff640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3e40d37c0 0x7ff3e414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e93e1640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3e415e170 0x7ff3e41573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 -- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3e415e170 msgr2=0x7ff3e41573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3e415e170 0x7ff3e41573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 -- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3e40d37c0 msgr2=0x7ff3e414d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3e40d37c0 0x7ff3e414d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 -- 172.21.15.6:0/233800455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e414d5f0 con 0x7ff3e40d3330 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e8be0640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e40d3330 0x7ff3e40ca1e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8000c00 tx=0x7ff3d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e17fa640 1 -- 172.21.15.6:0/233800455 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8040070 con 0x7ff3e40d3330 2024-03-18T14:42:07.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e17fa640 1 -- 172.21.15.6:0/233800455 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d803a910 con 0x7ff3e40d3330 2024-03-18T14:42:07.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e40cafc0 con 0x7ff3e40d3330 2024-03-18T14:42:07.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3e17fa640 1 -- 172.21.15.6:0/233800455 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8035040 con 0x7ff3e40d3330 2024-03-18T14:42:07.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.751+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e40cb580 con 0x7ff3e40d3330 2024-03-18T14:42:07.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.752+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e408afe0 con 0x7ff3e40d3330 2024-03-18T14:42:07.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.753+0000 7ff3e17fa640 1 -- 172.21.15.6:0/233800455 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff3d804b060 con 0x7ff3e40d3330 2024-03-18T14:42:07.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.753+0000 7ff3e17fa640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff3c806bc00 0x7ff3c806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:07.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.753+0000 7ff3e17fa640 1 -- 172.21.15.6:0/233800455 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff3d80ac490 con 0x7ff3e40d3330 2024-03-18T14:42:07.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.753+0000 7ff3e3fff640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff3c806bc00 0x7ff3c806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:07.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.754+0000 7ff3e3fff640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff3c806bc00 0x7ff3c806e0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4150cb0 tx=0x7ff3d0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:07.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.758+0000 7ff3e17fa640 1 -- 172.21.15.6:0/233800455 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff3d8031800 con 0x7ff3e40d3330 2024-03-18T14:42:07.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.942+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3e404f620 con 0x7ff3e40d3330 2024-03-18T14:42:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.943+0000 7ff3e17fa640 1 -- 172.21.15.6:0/233800455 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18418 (secure 0 0 0) 0x7ff3d807a5f0 con 0x7ff3e40d3330 2024-03-18T14:42:07.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:07.946 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:41:39.929559+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:07.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff3c806bc00 msgr2=0x7ff3c806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff3c806bc00 0x7ff3c806e0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4150cb0 tx=0x7ff3d0009190 comp rx=0 tx=0).stop 2024-03-18T14:42:07.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e40d3330 msgr2=0x7ff3e40ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:07.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e40d3330 0x7ff3e40ca1e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8000c00 tx=0x7ff3d80046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 shutdown_connections 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff3c806bc00 0x7ff3c806e0c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3e415e170 0x7ff3e41573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3e40d37c0 0x7ff3e414d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 --2- 172.21.15.6:0/233800455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3e40d3330 0x7ff3e40ca1e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 >> 172.21.15.6:0/233800455 conn(0x7ff3e406c900 msgr2=0x7ff3e40d3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 shutdown_connections 2024-03-18T14:42:07.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:07.945+0000 7ff3eae44640 1 -- 172.21.15.6:0/233800455 wait complete. 2024-03-18T14:42:07.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.057+0000 7f1a19113640 1 Processor -- start 2024-03-18T14:42:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.057+0000 7f1a19113640 1 -- start start 2024-03-18T14:42:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a19113640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a140ced00 0x7f1a14162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a19113640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140d3210 0x7f1a140d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a19113640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a140b9660 con 0x7f1a1415e020 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a19113640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1a140d2270 con 0x7f1a140ced00 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a19113640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1a140ce390 con 0x7f1a140d3210 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a1259c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a140ced00 0x7f1a14162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a1259c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a140ced00 0x7f1a14162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39992/0 (socket says 172.21.15.6:39992) 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a1259c640 1 -- 172.21.15.6:0/2491911079 learned_addr learned my addr 172.21.15.6:0/2491911079 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.058+0000 7f1a1359e640 1 --2- 172.21.15.6:0/2491911079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140d3210 0x7f1a140d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.059+0000 7f1a1259c640 1 -- 172.21.15.6:0/2491911079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140d3210 msgr2=0x7f1a140d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.059+0000 7f1a1259c640 1 --2- 172.21.15.6:0/2491911079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140d3210 0x7f1a140d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.059+0000 7f1a1259c640 1 -- 172.21.15.6:0/2491911079 >> v1:172.21.15.92:6789/0 conn(0x7f1a1415e020 legacy=0x7f1a140d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.059+0000 7f1a1259c640 1 -- 172.21.15.6:0/2491911079 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a140b9900 con 0x7f1a140ced00 2024-03-18T14:42:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.059+0000 7f1a11d9b640 1 -- 172.21.15.6:0/2491911079 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3835963441 0 0) 0x7f1a140b9660 con 0x7f1a1415e020 2024-03-18T14:42:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.059+0000 7f1a1359e640 1 --2- 172.21.15.6:0/2491911079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140d3210 0x7f1a140d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.059+0000 7f1a1259c640 1 --2- 172.21.15.6:0/2491911079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a140ced00 0x7f1a14162380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1a140b9830 tx=0x7f1a0802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb09e844a2e719e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a11d9b640 1 -- 172.21.15.6:0/2491911079 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a08035020 con 0x7f1a140ced00 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a11d9b640 1 -- 172.21.15.6:0/2491911079 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a08031640 con 0x7f1a140ced00 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a11d9b640 1 -- 172.21.15.6:0/2491911079 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0802d040 con 0x7f1a140ced00 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 -- 172.21.15.6:0/2491911079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a140ced00 msgr2=0x7f1a14162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 --2- 172.21.15.6:0/2491911079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a140ced00 0x7f1a14162380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1a140b9830 tx=0x7f1a0802b8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 -- 172.21.15.6:0/2491911079 shutdown_connections 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 --2- 172.21.15.6:0/2491911079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140d3210 0x7f1a140d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 --2- 172.21.15.6:0/2491911079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a140ced00 0x7f1a14162380 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 -- 172.21.15.6:0/2491911079 >> 172.21.15.6:0/2491911079 conn(0x7f1a14072ce0 msgr2=0x7f1a140730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 -- 172.21.15.6:0/2491911079 shutdown_connections 2024-03-18T14:42:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.060+0000 7f1a19113640 1 -- 172.21.15.6:0/2491911079 wait complete. 2024-03-18T14:42:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 Processor -- start 2024-03-18T14:42:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 -- start start 2024-03-18T14:42:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140ced00 0x7f1a1415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a140d3210 0x7f1a140c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a1415e020 0x7f1a140c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1a140cf2c0 con 0x7f1a1415e020 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1a140ce890 con 0x7f1a140d3210 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a19113640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1a140cf440 con 0x7f1a140ced00 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a1359e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a1415e020 0x7f1a140c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a1359e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a1415e020 0x7f1a140c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40006/0 (socket says 172.21.15.6:40006) 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a1359e640 1 -- 172.21.15.6:0/419910299 learned_addr learned my addr 172.21.15.6:0/419910299 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a1259c640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a140d3210 0x7f1a140c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.062+0000 7f1a12d9d640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140ced00 0x7f1a1415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a1359e640 1 -- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140ced00 msgr2=0x7f1a1415b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a1359e640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140ced00 0x7f1a1415b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a1359e640 1 -- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a140d3210 msgr2=0x7f1a140c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a1359e640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a140d3210 0x7f1a140c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a1359e640 1 -- 172.21.15.6:0/419910299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a140ca430 con 0x7f1a1415e020 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a1359e640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a1415e020 0x7f1a140c9ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00efc0 tx=0x7f1a0c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f19fb7fe640 1 -- 172.21.15.6:0/419910299 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c018030 con 0x7f1a1415e020 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f19fb7fe640 1 -- 172.21.15.6:0/419910299 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c00cde0 con 0x7f1a1415e020 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a14159380 con 0x7f1a1415e020 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f19fb7fe640 1 -- 172.21.15.6:0/419910299 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c003bb0 con 0x7f1a1415e020 2024-03-18T14:42:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.063+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a14153660 con 0x7f1a1415e020 2024-03-18T14:42:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.064+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a1408afe0 con 0x7f1a1415e020 2024-03-18T14:42:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.064+0000 7f19fb7fe640 1 -- 172.21.15.6:0/419910299 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1a0c005d10 con 0x7f1a1415e020 2024-03-18T14:42:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.065+0000 7f19fb7fe640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a0006bbd0 0x7f1a0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.065+0000 7f19fb7fe640 1 -- 172.21.15.6:0/419910299 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1a0c08b6a0 con 0x7f1a1415e020 2024-03-18T14:42:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.065+0000 7f1a12d9d640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a0006bbd0 0x7f1a0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.065+0000 7f1a12d9d640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a0006bbd0 0x7f1a0006e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1a04000f60 tx=0x7f1a04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.069+0000 7f19fb7fe640 1 -- 172.21.15.6:0/419910299 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1a0c022070 con 0x7f1a1415e020 2024-03-18T14:42:08.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.289+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a140d1940 con 0x7f1a1415e020 2024-03-18T14:42:08.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.289+0000 7f19fb7fe640 1 -- 172.21.15.6:0/419910299 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7f1a0c010040 con 0x7f1a1415e020 2024-03-18T14:42:08.292 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:08.292 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:42:08.292 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a0006bbd0 msgr2=0x7f1a0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a0006bbd0 0x7f1a0006e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1a04000f60 tx=0x7f1a04009040 comp rx=0 tx=0).stop 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a1415e020 msgr2=0x7f1a140c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a1415e020 0x7f1a140c9ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00efc0 tx=0x7f1a0c009f90 comp rx=0 tx=0).stop 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 shutdown_connections 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1a0006bbd0 0x7f1a0006e090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a1415e020 0x7f1a140c9ef0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a140d3210 0x7f1a140c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 --2- 172.21.15.6:0/419910299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a140ced00 0x7f1a1415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 >> 172.21.15.6:0/419910299 conn(0x7f1a14072ce0 msgr2=0x7f1a140cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 shutdown_connections 2024-03-18T14:42:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.291+0000 7f1a19113640 1 -- 172.21.15.6:0/419910299 wait complete. 2024-03-18T14:42:08.707 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T14:41:34.883167+0000', 'modified': '2024-03-18T14:41:56.958188+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6837/2327775013', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6835', 'nonce': 2327775013}, {'type': 'v1', 'addr': '172.21.15.92:6837', 'nonce': 2327775013}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-03-18T14:42:08.707 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T14:42:08.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-18T14:42:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.810+0000 7ff5b6fb2640 1 Processor -- start 2024-03-18T14:42:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.810+0000 7ff5b6fb2640 1 -- start start 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5b6fb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b01624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5b6fb2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff5b006caf0 0x7ff5b00acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5b6fb2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b00722c0 con 0x7ff5b006f9a0 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5b6fb2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff5b01629f0 con 0x7ff5b006caf0 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5b6fb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b015e3f0 con 0x7ff5b015e010 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5b554f640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff5b006caf0 0x7ff5b00acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5b4d4e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b01624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5affff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff5b006f9a0 0x7ff5b006c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:59144/0 (socket says 172.21.15.6:59144) 2024-03-18T14:42:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.811+0000 7ff5affff640 1 -- 172.21.15.6:0/3655446713 learned_addr learned my addr 172.21.15.6:0/3655446713 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/3655446713 >> v2:172.21.15.92:3300/0 conn(0x7ff5b006caf0 msgr2=0x7ff5b00acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/3655446713 >> v2:172.21.15.92:3300/0 conn(0x7ff5b006caf0 0x7ff5b00acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/3655446713 >> v1:172.21.15.92:6789/0 conn(0x7ff5b006f9a0 legacy=0x7ff5b006c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/3655446713 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b00725a0 con 0x7ff5b015e010 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5af7fe640 1 -- 172.21.15.6:0/3655446713 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200781138 0 0) 0x7ff5b00722c0 con 0x7ff5b006f9a0 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5b554f640 1 --2- 172.21.15.6:0/3655446713 >> v2:172.21.15.92:3300/0 conn(0x7ff5b006caf0 0x7ff5b00acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/3655446713 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b01624b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009d90 tx=0x7ff5a802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77083f5ba0d0ebf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.812+0000 7ff5af7fe640 1 -- 172.21.15.6:0/3655446713 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8039070 con 0x7ff5b015e010 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5af7fe640 1 -- 172.21.15.6:0/3655446713 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a802cce0 con 0x7ff5b015e010 2024-03-18T14:42:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5af7fe640 1 -- 172.21.15.6:0/3655446713 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8034660 con 0x7ff5b015e010 2024-03-18T14:42:08.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/3655446713 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 msgr2=0x7ff5b01624b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/3655446713 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b01624b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8009d90 tx=0x7ff5a802bf30 comp rx=0 tx=0).stop 2024-03-18T14:42:08.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/3655446713 shutdown_connections 2024-03-18T14:42:08.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/3655446713 >> v2:172.21.15.92:3300/0 conn(0x7ff5b006caf0 0x7ff5b00acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/3655446713 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b01624b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/3655446713 >> 172.21.15.6:0/3655446713 conn(0x7ff5b00d0110 msgr2=0x7ff5b00d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/3655446713 shutdown_connections 2024-03-18T14:42:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.813+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/3655446713 wait complete. 2024-03-18T14:42:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.814+0000 7ff5b6fb2640 1 Processor -- start 2024-03-18T14:42:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.814+0000 7ff5b6fb2640 1 -- start start 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b6fb2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5b006caf0 0x7ff5b01f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b6fb2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5b006f9a0 0x7ff5b015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b6fb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b00c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b6fb2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b0072b60 con 0x7ff5b006caf0 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b6fb2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b006e480 con 0x7ff5b006f9a0 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b6fb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b0072e40 con 0x7ff5b015e010 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b4d4e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5b006caf0 0x7ff5b01f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b4d4e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5b006caf0 0x7ff5b01f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40022/0 (socket says 172.21.15.6:40022) 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/592439830 learned_addr learned my addr 172.21.15.6:0/592439830 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5affff640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5b006f9a0 0x7ff5b015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.815+0000 7ff5b554f640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b00c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 msgr2=0x7ff5b00c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b00c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5b006f9a0 msgr2=0x7ff5b015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5b006f9a0 0x7ff5b015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/592439830 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b015ee60 con 0x7ff5b006caf0 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5b006caf0 0x7ff5b01f7b80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8037040 tx=0x7ff5a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5ad7fa640 1 -- 172.21.15.6:0/592439830 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8039050 con 0x7ff5b006caf0 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5ad7fa640 1 -- 172.21.15.6:0/592439830 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a803a040 con 0x7ff5b006caf0 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b015f140 con 0x7ff5b006caf0 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5ad7fa640 1 -- 172.21.15.6:0/592439830 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a803e610 con 0x7ff5b006caf0 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.816+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b01f7680 con 0x7ff5b006caf0 2024-03-18T14:42:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.817+0000 7ff5ad7fa640 1 -- 172.21.15.6:0/592439830 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff5a8047020 con 0x7ff5b006caf0 2024-03-18T14:42:08.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.818+0000 7ff5ad7fa640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff59c06bbd0 0x7ff59c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.818+0000 7ff5ad7fa640 1 -- 172.21.15.6:0/592439830 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff5a80acef0 con 0x7ff5b006caf0 2024-03-18T14:42:08.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.818+0000 7ff5affff640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff59c06bbd0 0x7ff59c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.818+0000 7ff5affff640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff59c06bbd0 0x7ff59c06e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00b5e80 tx=0x7ff5a4007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.819+0000 7ff5b9b95740 1 Processor -- start 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.819+0000 7ff5b9b95740 1 -- start start 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da3dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da6da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da3e160 0x55e78da6df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e78d9a6c80 con 0x55e78da3e160 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e78d94faa0 con 0x55e78da6b060 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x55e78d9a2d20 con 0x55e78da6ac80 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da3e160 0x55e78da6df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b4d4e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da3dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da3e160 0x55e78da6df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40038/0 (socket says 172.21.15.6:40038) 2024-03-18T14:42:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 -- 172.21.15.6:0/1177576845 learned_addr learned my addr 172.21.15.6:0/1177576845 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5affff640 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da6da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 -- 172.21.15.6:0/1177576845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 msgr2=0x55e78da3dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da3dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 -- 172.21.15.6:0/1177576845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 msgr2=0x55e78da6da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da6da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 -- 172.21.15.6:0/1177576845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e78d971ff0 con 0x55e78da3e160 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5b554f640 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da3e160 0x55e78da6df50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009b90 tx=0x7ff5a0011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc919a7b04636f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5967fc640 1 -- 172.21.15.6:0/1177576845 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0004020 con 0x55e78da3e160 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5967fc640 1 -- 172.21.15.6:0/1177576845 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a00041c0 con 0x55e78da3e160 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.820+0000 7ff5967fc640 1 -- 172.21.15.6:0/1177576845 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0003610 con 0x55e78da3e160 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1177576845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da3e160 msgr2=0x55e78da6df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da3e160 0x55e78da6df50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009b90 tx=0x7ff5a0011690 comp rx=0 tx=0).stop 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1177576845 shutdown_connections 2024-03-18T14:42:08.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da3e160 0x55e78da6df50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da6da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1177576845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da3dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1177576845 >> 172.21.15.6:0/1177576845 conn(0x55e78da3fe90 msgr2=0x55e78da88a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1177576845 shutdown_connections 2024-03-18T14:42:08.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.821+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1177576845 wait complete. 2024-03-18T14:42:08.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.822+0000 7ff5b9b95740 1 -- start start 2024-03-18T14:42:08.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.822+0000 7ff5b9b95740 1 Processor -- start 2024-03-18T14:42:08.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.822+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.822+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da70d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.822+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da89350 0x55e78da71250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.822+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e78da42530 con 0x55e78da6b060 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e78d962ef0 con 0x55e78da89350 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x55e78da44d70 con 0x55e78da6ac80 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b9b95740 10 client.0 _ll_register_callbacks cb 0x55e78d95ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da70d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da70d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40052/0 (socket says 172.21.15.6:40052) 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b4d4e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 -- 172.21.15.6:0/1283355762 learned_addr learned my addr 172.21.15.6:0/1283355762 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b554f640 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da89350 0x55e78da71250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 -- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 msgr2=0x55e78da707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 -- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da89350 msgr2=0x55e78da71250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da89350 0x55e78da71250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 -- 172.21.15.6:0/1283355762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e78d96fa30 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5affff640 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da70d10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4013620 tx=0x7ff5a4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecbe6721ddf50f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff58f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff58f7fe640 1 -- 172.21.15.6:0/1283355762 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4020700 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff58f7fe640 1 -- 172.21.15.6:0/1283355762 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff5a40208a0 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5a00041c0 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff58f7fe640 1 -- 172.21.15.6:0/1283355762 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7ff5a402e050 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b9b95740 10 client.4854 fetch_fsmap learned FSMap version 8 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.823+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55e78d6bb2d0 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff58f7fe640 1 -- 172.21.15.6:0/1283355762 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a00041c0 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff58f7fe640 1 -- 172.21.15.6:0/1283355762 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7ff5a4031020 con 0x55e78da6b060 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b9b95740 10 client.4854 fetch_fsmap finished waiting for FSMap version 8 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b9b95740 10 client.4854 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b9b95740 20 client.4854 populate_metadata read hostname 'smithi006' 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78daa4860 0x55e78daa6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b9b95740 4 client.4854 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 --> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e78d9b41c0 con 0x55e78daa4860 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78daa4860 0x55e78daa6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff58f7fe640 10 client.4854 ms_handle_connect on v2:172.21.15.92:6835/2327775013 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.824+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78daa4860 0x55e78daa6c50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f24eb532d17d446d server_cookie=90b671dd310360dd in_seq=0 out_seq=0 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff58f7fe640 1 -- 172.21.15.6:0/1283355762 <== mds.0 v2:172.21.15.92:6835/2327775013 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e78d9b41c0 con 0x55e78daa4860 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff58f7fe640 10 client.4854 handle_command_reply: tid=0 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 client.4854 shutdown 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 msgr2=0x55e78da70d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da70d10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4013620 tx=0x7ff5a4000ed0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 shutdown_connections 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78daa4860 0x55e78daa6c50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da89350 0x55e78da71250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da70d10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff58f7fe640 0 client.4854 ms_handle_reset on v2:172.21.15.92:6835/2327775013 2024-03-18T14:42:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/1283355762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da6ac80 0x55e78da707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 >> 172.21.15.6:0/1283355762 conn(0x55e78da3e160 msgr2=0x55e78da41d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 shutdown_connections 2024-03-18T14:42:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 1 -- 172.21.15.6:0/1283355762 wait complete. 2024-03-18T14:42:08.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.825+0000 7ff5b9b95740 20 client.4854 trim_cache size 0 max 16384 2024-03-18T14:42:08.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.843+0000 7ff5b9b95740 1 Processor -- start 2024-03-18T14:42:08.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.843+0000 7ff5b9b95740 1 -- start start 2024-03-18T14:42:08.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6ac80 0x55e78daabe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da9af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da9b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b0200840 con 0x55e78da6b060 2024-03-18T14:42:08.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b01ffca0 con 0x55e78da6ac80 2024-03-18T14:42:08.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b02001f0 con 0x55e78da89350 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da9af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da9af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40064/0 (socket says 172.21.15.6:40064) 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 -- 172.21.15.6:0/3704847027 learned_addr learned my addr 172.21.15.6:0/3704847027 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b554f640 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da9b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6ac80 0x55e78daabe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 -- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 msgr2=0x55e78da9b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da9b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 -- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6ac80 msgr2=0x55e78daabe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6ac80 0x55e78daabe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 -- 172.21.15.6:0/3704847027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e78d7856b0 con 0x55e78da6b060 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.844+0000 7ff5affff640 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da9af10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40017f0 tx=0x7ff5a4021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91915ad275805935 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff595ffb640 1 -- 172.21.15.6:0/3704847027 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4020d70 con 0x55e78da6b060 2024-03-18T14:42:08.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff595ffb640 1 -- 172.21.15.6:0/3704847027 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a4026070 con 0x55e78da6b060 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff595ffb640 1 -- 172.21.15.6:0/3704847027 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4020900 con 0x55e78da6b060 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 -- 172.21.15.6:0/3704847027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 msgr2=0x55e78da9af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da9af10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40017f0 tx=0x7ff5a4021000 comp rx=0 tx=0).stop 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 -- 172.21.15.6:0/3704847027 shutdown_connections 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da9b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6b060 0x55e78da9af10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/3704847027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6ac80 0x55e78daabe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 -- 172.21.15.6:0/3704847027 >> 172.21.15.6:0/3704847027 conn(0x55e78da3e160 msgr2=0x55e78da68730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 -- 172.21.15.6:0/3704847027 shutdown_connections 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.845+0000 7ff5b9b95740 1 -- 172.21.15.6:0/3704847027 wait complete. 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.846+0000 7ff5b9b95740 1 -- start start 2024-03-18T14:42:08.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.846+0000 7ff5b9b95740 1 Processor -- start 2024-03-18T14:42:08.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6ac80 0x55e78da73070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da735b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da67850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55e78da71850 con 0x55e78da6ac80 2024-03-18T14:42:08.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e78da6fdb0 con 0x55e78da6b060 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x55e78da9c990 con 0x55e78da89350 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 10 client.0 _ll_register_callbacks cb 0x7ff5a4004c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6ac80 0x55e78da73070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6ac80 0x55e78da73070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40076/0 (socket says 172.21.15.6:40076) 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/18114725 learned_addr learned my addr 172.21.15.6:0/18114725 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5affff640 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b554f640 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da67850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 msgr2=0x55e78da67850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da67850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 msgr2=0x55e78da735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da735b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 -- 172.21.15.6:0/18114725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e78da6e080 con 0x55e78da6ac80 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b4d4e640 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6ac80 0x55e78da73070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff5a802c540 tx=0x7ff5a80070f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21989ea2cf7f33a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff58f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff58f7fe640 1 -- 172.21.15.6:0/18114725 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8083740 con 0x55e78da6ac80 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff58f7fe640 1 -- 172.21.15.6:0/18114725 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff5a80776a0 con 0x55e78da6ac80 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5a4026070 con 0x55e78da6ac80 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff58f7fe640 1 -- 172.21.15.6:0/18114725 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7ff5a80a2050 con 0x55e78da6ac80 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 10 client.4860 fetch_fsmap learned FSMap version 8 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.847+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55e78da6b6e0 con 0x55e78da6ac80 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff58f7fe640 1 -- 172.21.15.6:0/18114725 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a4026070 con 0x55e78da6ac80 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff58f7fe640 1 -- 172.21.15.6:0/18114725 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7ff5a80832b0 con 0x55e78da6ac80 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5b9b95740 10 client.4860 fetch_fsmap finished waiting for FSMap version 8 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5b9b95740 10 client.4860 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5b9b95740 20 client.4860 populate_metadata read hostname 'smithi006' 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78da68100 0x55e78da89b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5b9b95740 4 client.4860 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:42:08.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 --> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] -- command(tid 0: {"prefix": "status"}) -- 0x55e78d9ae510 con 0x55e78da68100 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5affff640 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78da68100 0x55e78da89b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff58f7fe640 10 client.4860 ms_handle_connect on v2:172.21.15.92:6835/2327775013 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5affff640 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78da68100 0x55e78da89b40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9cccaa8e0e9cb37 server_cookie=83fd5af2ccc8b487 in_seq=0 out_seq=0 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff58f7fe640 1 -- 172.21.15.6:0/18114725 <== mds.0 v2:172.21.15.92:6835/2327775013 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55e78d9ae510 con 0x55e78da68100 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff58f7fe640 10 client.4860 handle_command_reply: tid=0 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.848+0000 7ff5b9b95740 1 client.4860 shutdown 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6ac80 msgr2=0x55e78da73070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6ac80 0x55e78da73070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff5a802c540 tx=0x7ff5a80070f0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 shutdown_connections 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:6835/2327775013,v1:172.21.15.92:6837/2327775013] conn(0x55e78da68100 0x55e78da89b40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e78da89350 0x55e78da67850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e78da6b060 0x55e78da735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff58f7fe640 0 client.4860 ms_handle_reset on v2:172.21.15.92:6835/2327775013 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 --2- 172.21.15.6:0/18114725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e78da6ac80 0x55e78da73070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 >> 172.21.15.6:0/18114725 conn(0x55e78da3e160 msgr2=0x55e78daabd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 shutdown_connections 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 1 -- 172.21.15.6:0/18114725 wait complete. 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b9b95740 20 client.4860 trim_cache size 0 max 16384 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a75a8f99-9e80-46f7-9da4-368af92e8063", 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4294, 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-18T14:42:08.858 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 33.311831292000001, 2024-03-18T14:42:08.859 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 8, 2024-03-18T14:42:08.859 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 25, 2024-03-18T14:42:08.859 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 25, 2024-03-18T14:42:08.859 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 39.896743772999997 2024-03-18T14:42:08.859 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff59c06bbd0 msgr2=0x7ff59c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff59c06bbd0 0x7ff59c06e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00b5e80 tx=0x7ff5a4007d50 comp rx=0 tx=0).stop 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5b006caf0 msgr2=0x7ff5b01f7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5b006caf0 0x7ff5b01f7b80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8037040 tx=0x7ff5a8004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 shutdown_connections 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff59c06bbd0 0x7ff59c06e090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff5b015e010 0x7ff5b00c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5b006f9a0 0x7ff5b015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 --2- 172.21.15.6:0/592439830 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5b006caf0 0x7ff5b01f7b80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 >> 172.21.15.6:0/592439830 conn(0x7ff5b00d0110 msgr2=0x7ff5b00afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 shutdown_connections 2024-03-18T14:42:08.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:08.849+0000 7ff5b6fb2640 1 -- 172.21.15.6:0/592439830 wait complete. 2024-03-18T14:42:08.863 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-18T14:42:08.873 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-18T14:42:08.873 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-18T14:42:08.873 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-18T14:42:08.873 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T14:42:08.873 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:08.873 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-18T14:42:08.873 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T14:42:08.873 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:08.873 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-18T14:42:08.873 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T14:42:08.873 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:08.874 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-18T14:42:08.874 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T14:42:08.874 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:08.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:08.874 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list 2024-03-18T14:42:08.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:08.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link delete ceph-brx 2024-03-18T14:42:08.958 INFO:teuthology.orchestra.run.smithi092.stderr:Cannot find device "ceph-brx" 2024-03-18T14:42:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:42:08.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:08.960 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:42:08.975 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-18T14:42:08.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:08.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T14:42:09.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:09.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-03-18T14:42:09.134 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-18T14:42:09.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:42:09.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.233+0000 7f9dab085640 1 Processor -- start 2024-03-18T14:42:09.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.233+0000 7f9dab085640 1 -- start start 2024-03-18T14:42:09.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9dab085640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9da40b9650 0x7f9da40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9dab085640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da40d0ff0 con 0x7f9da415e170 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9dab085640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da40b9ae0 con 0x7f9da40d0bc0 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9dab085640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9da40cef50 con 0x7f9da40b9650 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9da3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9da40b9650 0x7f9da40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9da3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9da40b9650 0x7f9da40d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40084/0 (socket says 172.21.15.6:40084) 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9da8e21640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9da415e170 0x7f9da40d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38622/0 (socket says 172.21.15.6:38622) 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.234+0000 7f9da3fff640 1 -- 172.21.15.6:0/1377527691 learned_addr learned my addr 172.21.15.6:0/1377527691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da3fff640 1 -- 172.21.15.6:0/1377527691 >> v1:172.21.15.92:6789/0 conn(0x7f9da40d0bc0 legacy=0x7f9da40b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da3fff640 1 -- 172.21.15.6:0/1377527691 >> v1:172.21.15.6:6789/0 conn(0x7f9da415e170 legacy=0x7f9da40d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da37fe640 1 -- 172.21.15.6:0/1377527691 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305059903 0 0) 0x7f9da40b9ae0 con 0x7f9da40d0bc0 2024-03-18T14:42:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da3fff640 1 -- 172.21.15.6:0/1377527691 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da40cf230 con 0x7f9da40b9650 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da3fff640 1 --2- 172.21.15.6:0/1377527691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40b9650 0x7f9da40d0680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009e50 tx=0x7f9d9402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8a480a0cd6f4f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da37fe640 1 -- 172.21.15.6:0/1377527691 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9402c960 con 0x7f9da40b9650 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da37fe640 1 -- 172.21.15.6:0/1377527691 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d9402cb00 con 0x7f9da40b9650 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.235+0000 7f9da37fe640 1 -- 172.21.15.6:0/1377527691 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d940343f0 con 0x7f9da40b9650 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.236+0000 7f9dab085640 1 -- 172.21.15.6:0/1377527691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40b9650 msgr2=0x7f9da40d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.236+0000 7f9dab085640 1 --2- 172.21.15.6:0/1377527691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40b9650 0x7f9da40d0680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009e50 tx=0x7f9d9402be40 comp rx=0 tx=0).stop 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.236+0000 7f9dab085640 1 -- 172.21.15.6:0/1377527691 shutdown_connections 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.236+0000 7f9dab085640 1 --2- 172.21.15.6:0/1377527691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40b9650 0x7f9da40d0680 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.236+0000 7f9dab085640 1 -- 172.21.15.6:0/1377527691 >> 172.21.15.6:0/1377527691 conn(0x7f9da406c900 msgr2=0x7f9da406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.236+0000 7f9dab085640 1 -- 172.21.15.6:0/1377527691 shutdown_connections 2024-03-18T14:42:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.236+0000 7f9dab085640 1 -- 172.21.15.6:0/1377527691 wait complete. 2024-03-18T14:42:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.237+0000 7f9dab085640 1 Processor -- start 2024-03-18T14:42:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.237+0000 7f9dab085640 1 -- start start 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9dab085640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9da40b9650 0x7f9da414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9dab085640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40d0bc0 0x7f9da41e3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9dab085640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9da415e170 0x7f9da40c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9dab085640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9da40d3a40 con 0x7f9da40d0bc0 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9dab085640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9da40d2e10 con 0x7f9da40b9650 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9dab085640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9da40d2820 con 0x7f9da415e170 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9da3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40d0bc0 0x7f9da41e3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9da3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40d0bc0 0x7f9da41e3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40086/0 (socket says 172.21.15.6:40086) 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9da3fff640 1 -- 172.21.15.6:0/2287417789 learned_addr learned my addr 172.21.15.6:0/2287417789 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9da8e21640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9da40b9650 0x7f9da414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.238+0000 7f9da9622640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9da415e170 0x7f9da40c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da3fff640 1 -- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9da415e170 msgr2=0x7f9da40c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da3fff640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9da415e170 0x7f9da40c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da3fff640 1 -- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9da40b9650 msgr2=0x7f9da414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da3fff640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9da40b9650 0x7f9da414b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da3fff640 1 -- 172.21.15.6:0/2287417789 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da41f3a40 con 0x7f9da40d0bc0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da3fff640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40d0bc0 0x7f9da41e3580 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d94036040 tx=0x7f9d94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da17fa640 1 -- 172.21.15.6:0/2287417789 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d94043020 con 0x7f9da40d0bc0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da17fa640 1 -- 172.21.15.6:0/2287417789 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d9403e450 con 0x7f9da40d0bc0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9da17fa640 1 -- 172.21.15.6:0/2287417789 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9403e840 con 0x7f9da40d0bc0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da41f3d20 con 0x7f9da40d0bc0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.239+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da41f42e0 con 0x7f9da40d0bc0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.240+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da408afe0 con 0x7f9da40d0bc0 2024-03-18T14:42:09.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.240+0000 7f9da17fa640 1 -- 172.21.15.6:0/2287417789 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9d94046020 con 0x7f9da40d0bc0 2024-03-18T14:42:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.241+0000 7f9da17fa640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9d8406bbd0 0x7f9d8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.241+0000 7f9da17fa640 1 -- 172.21.15.6:0/2287417789 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9d940ad570 con 0x7f9da40d0bc0 2024-03-18T14:42:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.241+0000 7f9da8e21640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9d8406bbd0 0x7f9d8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.241+0000 7f9da8e21640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9d8406bbd0 0x7f9d8406e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c000f60 tx=0x7f9d8c007e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.245+0000 7f9da17fa640 1 -- 172.21.15.6:0/2287417789 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9d9404d030 con 0x7f9da40d0bc0 2024-03-18T14:42:09.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.467+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9da40b86f0 con 0x7f9da40d0bc0 2024-03-18T14:42:09.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.468+0000 7f9da17fa640 1 -- 172.21.15.6:0/2287417789 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f9d9407b760 con 0x7f9da40d0bc0 2024-03-18T14:42:09.470 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T14:42:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.469+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9d8406bbd0 msgr2=0x7f9d8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.469+0000 7f9dab085640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9d8406bbd0 0x7f9d8406e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c000f60 tx=0x7f9d8c007e20 comp rx=0 tx=0).stop 2024-03-18T14:42:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.469+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40d0bc0 msgr2=0x7f9da41e3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.469+0000 7f9dab085640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40d0bc0 0x7f9da41e3580 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d94036040 tx=0x7f9d94004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.469+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 shutdown_connections 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.469+0000 7f9dab085640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9d8406bbd0 0x7f9d8406e090 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.470+0000 7f9dab085640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9da415e170 0x7f9da40c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.470+0000 7f9dab085640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9da40d0bc0 0x7f9da41e3580 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.470+0000 7f9dab085640 1 --2- 172.21.15.6:0/2287417789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9da40b9650 0x7f9da414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.470+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 >> 172.21.15.6:0/2287417789 conn(0x7f9da406c900 msgr2=0x7f9da40b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.470+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 shutdown_connections 2024-03-18T14:42:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09.470+0000 7f9dab085640 1 -- 172.21.15.6:0/2287417789 wait complete. 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:42:09.483 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:42:09.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:09.484 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:42:09.498 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:09.498 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:09.498 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:09.498 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:09.498 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42636sec preferred_lft 42636sec 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:09.499 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:09.500 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:09.500 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:42:09.500 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T14:42:09.500 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:09.500 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:09.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-03-18T14:42:09.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-03-18T14:42:09.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-03-18T14:42:09.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T14:42:09.500 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:09.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:09.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:09.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:09.646 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:42:09.671 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:42:09.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:09.672 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:42:09.728 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:42:09.728 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:42:09.728 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:42:09.728 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:42:09.728 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:42:09.728 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:09.729 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:09.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:42:09.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:42:09.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:42:09.729 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:09.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:09.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:09.864 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:42:09.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:09.879 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-18T14:42:09.932 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:09.932 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:09.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:09.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T14:42:09.932 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:10.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:10.054 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T14:42:10.054 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:10.054 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:10.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T14:42:10.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T14:42:10.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T14:42:10.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T14:42:10.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T14:42:10.055 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:10.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:10.178 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:10.178 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:10.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:42:10.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T14:42:10.178 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:10.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:10.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:10.259 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T14:42:10.259 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:10.259 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:10.273 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:42:10.273 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:10.274 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:10.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:42:10.394 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:42:10.450 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:42:10.451 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:42:10.452 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:42:10.452 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:42:10.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:42:10.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:10.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:10.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:10.452 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:42:10.504 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:42:10.505 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T14:42:10.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:42:10.574 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:10.629 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:42:10.630 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:42:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:42:10.632 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:42:10.632 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:42:10.632 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:42:10.632 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:42:10.632 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:10.632 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:10.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:10.632 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:42:10.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:2024-03-18T14:42:10.630+0000 7f0c530bb480 -1 init, newargv = 0x55b5b004d140 newargc=15 2024-03-18T14:42:10.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[48234]: starting ceph client 2024-03-18T14:42:10.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[48234]: starting fuse 2024-03-18T14:42:10.648 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:42:10.648 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T14:42:10.648 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:10.649 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.0")) 2024-03-18T14:42:10.650 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:42:10.768 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.0.48234.asok 2024-03-18T14:42:10.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:10.778 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48234.asok 2024-03-18T14:42:10.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:10.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48234.asok status 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "0", 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "48234", 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:42:10.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4875, 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4875 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3858149189 2024-03-18T14:42:10.917 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3858149189 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.4875 192.168.0.1:0/3858149189", 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/3858149189", 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 8, 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 25, 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:42:10.918 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:42:10.919 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:42:10.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:42:11.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.024+0000 7f2af2990640 1 Processor -- start 2024-03-18T14:42:11.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.024+0000 7f2af2990640 1 -- start start 2024-03-18T14:42:11.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af2990640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2aec15e020 0x7f2aec162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:11.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af2990640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af2990640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aec0b9660 con 0x7f2aec0ced00 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af2990640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2aec0d2270 con 0x7f2aec15e020 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af2990640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2aec0ce390 con 0x7f2aec0d3210 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af0f2d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2aebfff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2aec15e020 0x7f2aec162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2aeb7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2aec0ced00 0x7f2aec0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:59156/0 (socket says 172.21.15.6:59156) 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af0f2d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40582/0 (socket says 172.21.15.6:40582) 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2aeb7fe640 1 -- 172.21.15.6:0/152853189 learned_addr learned my addr 172.21.15.6:0/152853189 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af0f2d640 1 -- 172.21.15.6:0/152853189 >> v2:172.21.15.92:3300/0 conn(0x7f2aec15e020 msgr2=0x7f2aec162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af0f2d640 1 --2- 172.21.15.6:0/152853189 >> v2:172.21.15.92:3300/0 conn(0x7f2aec15e020 0x7f2aec162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af0f2d640 1 -- 172.21.15.6:0/152853189 >> v1:172.21.15.92:6789/0 conn(0x7f2aec0ced00 legacy=0x7f2aec0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:11.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2aeaffd640 1 -- 172.21.15.6:0/152853189 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333444077 0 0) 0x7f2aec0b9660 con 0x7f2aec0ced00 2024-03-18T14:42:11.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2af0f2d640 1 -- 172.21.15.6:0/152853189 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec0b9900 con 0x7f2aec0d3210 2024-03-18T14:42:11.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.025+0000 7f2aebfff640 1 --2- 172.21.15.6:0/152853189 >> v2:172.21.15.92:3300/0 conn(0x7f2aec15e020 0x7f2aec162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:11.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.026+0000 7f2af0f2d640 1 --2- 172.21.15.6:0/152853189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec0d3c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0b9830 tx=0x7f2ae402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf0fc11bca61f744 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:11.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.026+0000 7f2aeaffd640 1 -- 172.21.15.6:0/152853189 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4039070 con 0x7f2aec0d3210 2024-03-18T14:42:11.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.026+0000 7f2aeaffd640 1 -- 172.21.15.6:0/152853189 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae402cbb0 con 0x7f2aec0d3210 2024-03-18T14:42:11.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.026+0000 7f2aeaffd640 1 -- 172.21.15.6:0/152853189 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4034440 con 0x7f2aec0d3210 2024-03-18T14:42:11.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 -- 172.21.15.6:0/152853189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 msgr2=0x7f2aec0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:11.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 --2- 172.21.15.6:0/152853189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec0d3c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0b9830 tx=0x7f2ae402be00 comp rx=0 tx=0).stop 2024-03-18T14:42:11.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 -- 172.21.15.6:0/152853189 shutdown_connections 2024-03-18T14:42:11.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 --2- 172.21.15.6:0/152853189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec0d3c30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 --2- 172.21.15.6:0/152853189 >> v2:172.21.15.92:3300/0 conn(0x7f2aec15e020 0x7f2aec162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 -- 172.21.15.6:0/152853189 >> 172.21.15.6:0/152853189 conn(0x7f2aec072ce0 msgr2=0x7f2aec0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:11.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 -- 172.21.15.6:0/152853189 shutdown_connections 2024-03-18T14:42:11.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.027+0000 7f2af2990640 1 -- 172.21.15.6:0/152853189 wait complete. 2024-03-18T14:42:11.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.028+0000 7f2af2990640 1 Processor -- start 2024-03-18T14:42:11.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.028+0000 7f2af2990640 1 -- start start 2024-03-18T14:42:11.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2af2990640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aec0ced00 0x7f2aec153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2af2990640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2af2990640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2aec15e020 0x7f2aec0b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2af2990640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0cf2c0 con 0x7f2aec0ced00 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2af2990640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0ce890 con 0x7f2aec15e020 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2af2990640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2aec0cf440 con 0x7f2aec0d3210 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2aebfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aec0ced00 0x7f2aec153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2aebfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aec0ced00 0x7f2aec153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40092/0 (socket says 172.21.15.6:40092) 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2aebfff640 1 -- 172.21.15.6:0/692424782 learned_addr learned my addr 172.21.15.6:0/692424782 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2aeb7fe640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.029+0000 7f2af0f2d640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2aec15e020 0x7f2aec0b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2aebfff640 1 -- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 msgr2=0x7f2aec152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2aebfff640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2aebfff640 1 -- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2aec15e020 msgr2=0x7f2aec0b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2aebfff640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2aec15e020 0x7f2aec0b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2aebfff640 1 -- 172.21.15.6:0/692424782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec15e970 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2aeb7fe640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2aebfff640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aec0ced00 0x7f2aec153a30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000e960 tx=0x7f2ae000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2ae8ff9640 1 -- 172.21.15.6:0/692424782 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0017070 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2ae8ff9640 1 -- 172.21.15.6:0/692424782 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae000ca00 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2ae8ff9640 1 -- 172.21.15.6:0/692424782 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae000cdd0 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aec15ec50 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.030+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aec15f4f0 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.031+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aec15ee20 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.031+0000 7f2ae8ff9640 1 -- 172.21.15.6:0/692424782 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2ae001a070 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.032+0000 7f2ae8ff9640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ad806bbd0 0x7f2ad806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.032+0000 7f2ae8ff9640 1 -- 172.21.15.6:0/692424782 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2ae008aeb0 con 0x7f2aec0ced00 2024-03-18T14:42:11.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.032+0000 7f2aeb7fe640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ad806bbd0 0x7f2ad806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:11.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.032+0000 7f2aeb7fe640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ad806bbd0 0x7f2ad806e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2aec154a30 tx=0x7f2adc007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:11.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.036+0000 7f2ae8ff9640 1 -- 172.21.15.6:0/692424782 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2ae00590a0 con 0x7f2aec0ced00 2024-03-18T14:42:11.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.261+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2aec08afe0 con 0x7f2aec0ced00 2024-03-18T14:42:11.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.262+0000 7f2ae8ff9640 1 -- 172.21.15.6:0/692424782 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f2ae005d140 con 0x7f2aec0ced00 2024-03-18T14:42:11.264 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T14:42:11.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ad806bbd0 msgr2=0x7f2ad806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:11.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ad806bbd0 0x7f2ad806e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2aec154a30 tx=0x7f2adc007d50 comp rx=0 tx=0).stop 2024-03-18T14:42:11.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aec0ced00 msgr2=0x7f2aec153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aec0ced00 0x7f2aec153a30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000e960 tx=0x7f2ae000ee20 comp rx=0 tx=0).stop 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 shutdown_connections 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ad806bbd0 0x7f2ad806e090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2aec15e020 0x7f2aec0b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.263+0000 7f2af2990640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2aec0d3210 0x7f2aec152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.264+0000 7f2af2990640 1 --2- 172.21.15.6:0/692424782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aec0ced00 0x7f2aec153a30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.264+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 >> 172.21.15.6:0/692424782 conn(0x7f2aec072ce0 msgr2=0x7f2aec0b8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.264+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 shutdown_connections 2024-03-18T14:42:11.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11.264+0000 7f2af2990640 1 -- 172.21.15.6:0/692424782 wait complete. 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:42:11.277 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-18T14:42:11.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:11.278 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:42:11.292 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:11.292 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:42:11.293 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42634sec preferred_lft 42634sec 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c41d:adff:fe29:e32/64 scope link 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:11.294 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::d035:11ff:fe58:6e4e/64 scope link 2024-03-18T14:42:11.295 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:11.295 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:42:11.357 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:42:11.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:11.359 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:42:11.414 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:42:11.414 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:42:11.414 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:42:11.414 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:42:11.414 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:42:11.415 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:11.415 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:11.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:42:11.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:42:11.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:42:11.415 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:11.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:11.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:11.548 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:42:11.562 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:42:11.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:11.563 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-18T14:42:11.618 INFO:teuthology.orchestra.run.smithi006.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-18T14:42:11.618 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:11.618 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:11.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:42:11.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-18T14:42:11.619 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:11.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:11.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:11.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-18T14:42:11.765 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 3a:83:0c:bb:89:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::3883:cff:febb:8916/64 scope link 2024-03-18T14:42:11.766 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:11.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 3a:83:0c:bb:89:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T14:42:11.834 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.835 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::3883:cff:febb:8916/64 scope link 2024-03-18T14:42:11.835 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:11.835 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-18T14:42:11.836 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:11.999 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:11.999 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:42:11.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.1 up 2024-03-18T14:42:11.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.1 master ceph-brx 2024-03-18T14:42:11.999 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:42:12.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:12.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:12.080 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-18T14:42:12.080 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:12.080 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:12.094 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-18T14:42:12.094 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:12.094 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:12.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:42:12.216 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:12.271 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:42:12.271 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:42:12.271 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:42:12.271 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:42:12.271 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:42:12.272 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:12.273 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:12.274 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:42:12.274 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:42:12.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:12.274 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:42:12.326 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:42:12.326 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-18T14:42:12.326 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-18T14:42:12.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:42:12.395 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:42:12.450 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:42:12.451 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:42:12.452 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:42:12.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:12.453 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:42:12.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi006.stderr:2024-03-18T14:42:12.451+0000 7f9ea883b480 -1 init, newargv = 0x557b808e2440 newargc=15 2024-03-18T14:42:12.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi006.stderr:ceph-fuse[48732]: starting ceph client 2024-03-18T14:42:12.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi006.stderr:ceph-fuse[48732]: starting fuse 2024-03-18T14:42:12.468 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:42:12.468 INFO:teuthology.orchestra.run.smithi006.stdout:44 2024-03-18T14:42:12.468 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-18T14:42:12.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:42:12.469 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.2")) 2024-03-18T14:42:12.470 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:42:12.589 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.2.48732.asok 2024-03-18T14:42:12.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:12.599 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.48732.asok 2024-03-18T14:42:12.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:12.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.48732.asok status 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "2", 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "48732", 2024-03-18T14:42:12.737 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4884, 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4884 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:42:12.738 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.2:0", 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3203287176 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.2:0", 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3203287176 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.4884 192.168.0.2:0/3203287176", 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.2:0/3203287176", 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 8, 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 25, 2024-03-18T14:42:12.739 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:42:12.740 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:42:12.740 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:42:12.740 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:42:12.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:42:12.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.844+0000 7fbacdbf4640 1 Processor -- start 2024-03-18T14:42:12.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.844+0000 7fbacdbf4640 1 -- start start 2024-03-18T14:42:12.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbacdbf4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80d17f0 0x7fbac80b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:12.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbacdbf4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbac80cc990 0x7fbac80cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:12.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbacdbf4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbac80b1eb0 con 0x7fbac80d05d0 2024-03-18T14:42:12.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbacdbf4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbac80b2190 con 0x7fbac80cc990 2024-03-18T14:42:12.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbacdbf4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80b2470 con 0x7fbac80d17f0 2024-03-18T14:42:12.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac7fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbac80cc990 0x7fbac80cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:12.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac6ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbac80d05d0 0x7fbac80d1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38626/0 (socket says 172.21.15.6:38626) 2024-03-18T14:42:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac6ffd640 1 -- 172.21.15.6:0/2325941691 learned_addr learned my addr 172.21.15.6:0/2325941691 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac77fe640 1 --2- 172.21.15.6:0/2325941691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80d17f0 0x7fbac80b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac7fff640 1 -- 172.21.15.6:0/2325941691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80d17f0 msgr2=0x7fbac80b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac7fff640 1 --2- 172.21.15.6:0/2325941691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80d17f0 0x7fbac80b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac7fff640 1 -- 172.21.15.6:0/2325941691 >> v1:172.21.15.6:6789/0 conn(0x7fbac80d05d0 legacy=0x7fbac80d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac7fff640 1 -- 172.21.15.6:0/2325941691 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac80b2750 con 0x7fbac80cc990 2024-03-18T14:42:12.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.845+0000 7fbac67fc640 1 -- 172.21.15.6:0/2325941691 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1624977435 0 0) 0x7fbac80b1eb0 con 0x7fbac80d05d0 2024-03-18T14:42:12.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.846+0000 7fbac7fff640 1 --2- 172.21.15.6:0/2325941691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80cc990 0x7fbac80cce20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009d90 tx=0x7fbac002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e85fcf283450a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.846+0000 7fbac67fc640 1 -- 172.21.15.6:0/2325941691 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0039070 con 0x7fbac80cc990 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.846+0000 7fbac67fc640 1 -- 172.21.15.6:0/2325941691 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac002cc50 con 0x7fbac80cc990 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.846+0000 7fbac67fc640 1 -- 172.21.15.6:0/2325941691 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0034440 con 0x7fbac80cc990 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 -- 172.21.15.6:0/2325941691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80cc990 msgr2=0x7fbac80cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/2325941691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80cc990 0x7fbac80cce20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009d90 tx=0x7fbac002bea0 comp rx=0 tx=0).stop 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 -- 172.21.15.6:0/2325941691 shutdown_connections 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/2325941691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80cc990 0x7fbac80cce20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/2325941691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80d17f0 0x7fbac80b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 -- 172.21.15.6:0/2325941691 >> 172.21.15.6:0/2325941691 conn(0x7fbac808b130 msgr2=0x7fbac808b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 -- 172.21.15.6:0/2325941691 shutdown_connections 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.847+0000 7fbacdbf4640 1 -- 172.21.15.6:0/2325941691 wait complete. 2024-03-18T14:42:12.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.848+0000 7fbacdbf4640 1 Processor -- start 2024-03-18T14:42:12.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.848+0000 7fbacdbf4640 1 -- start start 2024-03-18T14:42:12.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbacdbf4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80cc990 0x7fbac81f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbacdbf4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80d05d0 0x7fbac815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbacdbf4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80d17f0 0x7fbac80ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbacdbf4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80d0a30 con 0x7fbac80d05d0 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbacdbf4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbac815c080 con 0x7fbac80d17f0 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbacdbf4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80cd360 con 0x7fbac80cc990 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80d05d0 0x7fbac815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80d05d0 0x7fbac815ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40116/0 (socket says 172.21.15.6:40116) 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 -- 172.21.15.6:0/550702656 learned_addr learned my addr 172.21.15.6:0/550702656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac77fe640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80cc990 0x7fbac81f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac7fff640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80d17f0 0x7fbac80ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:12.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 -- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80cc990 msgr2=0x7fbac81f5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80cc990 0x7fbac81f5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 -- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80d17f0 msgr2=0x7fbac80ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80d17f0 0x7fbac80ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.849+0000 7fbac6ffd640 1 -- 172.21.15.6:0/550702656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac815d9c0 con 0x7fbac80d05d0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.850+0000 7fbac6ffd640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80d05d0 0x7fbac815ccb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00c4d0 tx=0x7fbabc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.850+0000 7fbaaffff640 1 -- 172.21.15.6:0/550702656 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc018070 con 0x7fbac80d05d0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.850+0000 7fbaaffff640 1 -- 172.21.15.6:0/550702656 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbabc007500 con 0x7fbac80d05d0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.850+0000 7fbaaffff640 1 -- 172.21.15.6:0/550702656 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc004a20 con 0x7fbac80d05d0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.850+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac815d400 con 0x7fbac80d05d0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.850+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac80c6aa0 con 0x7fbac80d05d0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.851+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac809fa80 con 0x7fbac80d05d0 2024-03-18T14:42:12.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.851+0000 7fbaaffff640 1 -- 172.21.15.6:0/550702656 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbabc01b070 con 0x7fbac80d05d0 2024-03-18T14:42:12.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.852+0000 7fbaaffff640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbab406bbd0 0x7fbab406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:12.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.852+0000 7fbaaffff640 1 -- 172.21.15.6:0/550702656 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbabc089f70 con 0x7fbac80d05d0 2024-03-18T14:42:12.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.852+0000 7fbac77fe640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbab406bbd0 0x7fbab406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:12.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.852+0000 7fbac77fe640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbab406bbd0 0x7fbab406e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009aa0 tx=0x7fbab8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:12.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:12.856+0000 7fbaaffff640 1 -- 172.21.15.6:0/550702656 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbabc058160 con 0x7fbac80d05d0 2024-03-18T14:42:13.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.077+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbac80b79a0 con 0x7fbac80d05d0 2024-03-18T14:42:13.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.078+0000 7fbaaffff640 1 -- 172.21.15.6:0/550702656 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fbabc05c200 con 0x7fbac80d05d0 2024-03-18T14:42:13.080 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbab406bbd0 msgr2=0x7fbab406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbab406bbd0 0x7fbab406e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009aa0 tx=0x7fbab8009190 comp rx=0 tx=0).stop 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80d05d0 msgr2=0x7fbac815ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80d05d0 0x7fbac815ccb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00c4d0 tx=0x7fbabc00c990 comp rx=0 tx=0).stop 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 shutdown_connections 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbab406bbd0 0x7fbab406e090 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbac80d17f0 0x7fbac80ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbac80d05d0 0x7fbac815ccb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 --2- 172.21.15.6:0/550702656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbac80cc990 0x7fbac81f5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 >> 172.21.15.6:0/550702656 conn(0x7fbac808b130 msgr2=0x7fbac80cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 shutdown_connections 2024-03-18T14:42:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:13.080+0000 7fbacdbf4640 1 -- 172.21.15.6:0/550702656 wait complete. 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi092.front.sepia.ceph.com 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:42:13.093 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-18T14:42:13.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:13.094 DEBUG:teuthology.orchestra.run.smithi092:> ip addr 2024-03-18T14:42:13.108 INFO:teuthology.orchestra.run.smithi092.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b2 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f0 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b3 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f1 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:13.109 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:44 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:13.110 INFO:teuthology.orchestra.run.smithi092.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:42:13.110 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:45 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:13.110 INFO:teuthology.orchestra.run.smithi092.stdout: inet 172.21.15.92/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T14:42:13.110 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft 42648sec preferred_lft 42648sec 2024-03-18T14:42:13.110 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::ec4:7aff:fe8f:cd45/64 scope link 2024-03-18T14:42:13.110 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:13.110 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T14:42:13.110 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:13.110 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:13.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link add name ceph-brx type bridge 2024-03-18T14:42:13.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip addr flush dev ceph-brx 2024-03-18T14:42:13.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set ceph-brx up 2024-03-18T14:42:13.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T14:42:13.111 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:13.195 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:13.309 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:13.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:13.316 DEBUG:teuthology.orchestra.run.smithi092:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:42:13.343 INFO:teuthology.orchestra.run.smithi092.stdout:1 2024-03-18T14:42:13.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:13.344 DEBUG:teuthology.orchestra.run.smithi092:> route 2024-03-18T14:42:13.414 INFO:teuthology.orchestra.run.smithi092.stdout:Kernel IP routing table 2024-03-18T14:42:13.414 INFO:teuthology.orchestra.run.smithi092.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:42:13.414 INFO:teuthology.orchestra.run.smithi092.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T14:42:13.414 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T14:42:13.414 INFO:teuthology.orchestra.run.smithi092.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:42:13.414 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:13.414 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:13.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T14:42:13.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T14:42:13.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T14:42:13.415 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:13.501 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:13.696 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:13.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:13.703 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list 2024-03-18T14:42:13.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:13.719 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list-id 2024-03-18T14:42:13.774 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:13.775 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:13.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:13.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-18T14:42:13.775 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:13.862 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:13.893 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:13.900 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-18T14:42:13.900 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:13.949 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:14.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:14.047 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:14.047 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:14.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set brx.0 up 2024-03-18T14:42:14.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T14:42:14.048 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:14.099 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:14.127 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:14.133 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-18T14:42:14.133 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:14.133 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:14.148 INFO:teuthology.orchestra.run.smithi092.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-18T14:42:14.148 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:14.149 DEBUG:teuthology.orchestra.run.smithi092:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:14.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo modprobe fuse 2024-03-18T14:42:14.271 DEBUG:teuthology.orchestra.run.smithi092:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/proc 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/sys 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/dev 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/security 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/shm 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/pts 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/run 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/cgroup 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/pstore 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/bpf 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/ 2024-03-18T14:42:14.327 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/selinux 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/debug 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/mqueue 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/hugepages 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/tracing 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/config 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/run/user/1000 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-4 2024-03-18T14:42:14.328 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-5 2024-03-18T14:42:14.329 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-6 2024-03-18T14:42:14.329 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-7 2024-03-18T14:42:14.329 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns 2024-03-18T14:42:14.329 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:14.329 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:14.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:14.330 DEBUG:teuthology.orchestra.run.smithi092:> ls /sys/fs/fuse/connections 2024-03-18T14:42:14.383 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:42:14.383 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-18T14:42:14.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo modprobe fuse 2024-03-18T14:42:14.453 DEBUG:teuthology.orchestra.run.smithi092:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:14.508 INFO:teuthology.orchestra.run.smithi092.stdout:/proc 2024-03-18T14:42:14.508 INFO:teuthology.orchestra.run.smithi092.stdout:/sys 2024-03-18T14:42:14.508 INFO:teuthology.orchestra.run.smithi092.stdout:/dev 2024-03-18T14:42:14.508 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/security 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/shm 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/pts 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/run 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/cgroup 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/pstore 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/bpf 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/ 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/selinux 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/debug 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/mqueue 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/hugepages 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/tracing 2024-03-18T14:42:14.509 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/config 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/run/user/1000 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-4 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-5 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-6 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-7 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:14.510 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:14.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:14.511 DEBUG:teuthology.orchestra.run.smithi092:> ls /sys/fs/fuse/connections 2024-03-18T14:42:14.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi092.stderr:2024-03-18T14:42:14.510+0000 7fed6503c480 -1 init, newargv = 0x563647617700 newargc=15 2024-03-18T14:42:14.513 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi092.stderr:ceph-fuse[44164]: starting ceph client 2024-03-18T14:42:14.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi092.stderr:ceph-fuse[44164]: starting fuse 2024-03-18T14:42:14.527 INFO:teuthology.orchestra.run.smithi092.stdout:42 2024-03-18T14:42:14.528 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> import glob 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> import re 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> import os 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> import subprocess 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> def _find_admin_socket(client_name): 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> files = glob.glob(asok_path) 2024-03-18T14:42:14.528 DEBUG:teuthology.orchestra.run.smithi092:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> # Given a non-glob path, it better be there 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> if "*" not in asok_path: 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> assert(len(files) == 1) 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> return files[0] 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> for f in files: 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> contents = proc_f.read() 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> if mountpoint in contents: 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> return f 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:14.529 DEBUG:teuthology.orchestra.run.smithi092:> print(_find_admin_socket("client.1")) 2024-03-18T14:42:14.530 DEBUG:teuthology.orchestra.run.smithi092:> ' 2024-03-18T14:42:14.649 INFO:teuthology.orchestra.run.smithi092.stdout:/var/run/ceph/ceph-client.1.44164.asok 2024-03-18T14:42:14.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:14.661 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44164.asok 2024-03-18T14:42:14.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:14.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44164.asok status 2024-03-18T14:42:14.798 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-18T14:42:14.798 INFO:teuthology.orchestra.run.smithi092.stdout: "metadata": { 2024-03-18T14:42:14.798 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:42:14.798 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "entity_id": "1", 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "hostname": "smithi092", 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "pid": "44164", 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "root": "/" 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "dentry_count": 0, 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "dentry_pinned_count": 0, 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "id": 4894, 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "inst": { 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "name": { 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "client", 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: "num": 4894 2024-03-18T14:42:14.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": { 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 4115409824 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": { 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 4115409824 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "inst_str": "client.4894 192.168.0.1:0/4115409824", 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "addr_str": "192.168.0.1:0/4115409824", 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "inode_count": 1, 2024-03-18T14:42:14.800 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_epoch": 8, 2024-03-18T14:42:14.801 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_epoch": 25, 2024-03-18T14:42:14.801 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:42:14.801 INFO:teuthology.orchestra.run.smithi092.stdout: "blocklisted": false, 2024-03-18T14:42:14.801 INFO:teuthology.orchestra.run.smithi092.stdout: "fs_name": "cephfs" 2024-03-18T14:42:14.801 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-18T14:42:14.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:42:14.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.917+0000 7f194fdbc640 1 Processor -- start 2024-03-18T14:42:14.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194fdbc640 1 -- start start 2024-03-18T14:42:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194fdbc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f194815bd40 0x7f19480d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194fdbc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19480cfea0 0x7f19480cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194fdbc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19480cc870 con 0x7f19480d1d90 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194fdbc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f19480d0500 con 0x7f194815bd40 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194fdbc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f19480d07e0 con 0x7f19480cfea0 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194db58640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f194815bd40 0x7f19480d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194d357640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19480d1d90 0x7f19480cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54962/0 (socket says 172.21.15.6:54962) 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.918+0000 7f194d357640 1 -- 172.21.15.6:0/2663012677 learned_addr learned my addr 172.21.15.6:0/2663012677 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.919+0000 7f194e359640 1 --2- 172.21.15.6:0/2663012677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19480cfea0 0x7f19480cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.919+0000 7f194db58640 1 -- 172.21.15.6:0/2663012677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19480cfea0 msgr2=0x7f19480cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.919+0000 7f194db58640 1 --2- 172.21.15.6:0/2663012677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19480cfea0 0x7f19480cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.919+0000 7f194db58640 1 -- 172.21.15.6:0/2663012677 >> v1:172.21.15.6:6789/0 conn(0x7f19480d1d90 legacy=0x7f19480cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.919+0000 7f194cb56640 1 -- 172.21.15.6:0/2663012677 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2541642941 0 0) 0x7f19480cc870 con 0x7f19480d1d90 2024-03-18T14:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.919+0000 7f194db58640 1 -- 172.21.15.6:0/2663012677 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19480ccb10 con 0x7f194815bd40 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194db58640 1 --2- 172.21.15.6:0/2663012677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194815bd40 0x7f19480d1850 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f19400097a0 tx=0x7f194002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1035c2c8323ad0e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194cb56640 1 -- 172.21.15.6:0/2663012677 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940035020 con 0x7f194815bd40 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194cb56640 1 -- 172.21.15.6:0/2663012677 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1940031830 con 0x7f194815bd40 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194cb56640 1 -- 172.21.15.6:0/2663012677 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940031c00 con 0x7f194815bd40 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 -- 172.21.15.6:0/2663012677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194815bd40 msgr2=0x7f19480d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2663012677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194815bd40 0x7f19480d1850 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f19400097a0 tx=0x7f194002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 -- 172.21.15.6:0/2663012677 shutdown_connections 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2663012677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19480cfea0 0x7f19480cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2663012677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194815bd40 0x7f19480d1850 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 -- 172.21.15.6:0/2663012677 >> 172.21.15.6:0/2663012677 conn(0x7f194809ae20 msgr2=0x7f194808b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 -- 172.21.15.6:0/2663012677 shutdown_connections 2024-03-18T14:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.920+0000 7f194fdbc640 1 -- 172.21.15.6:0/2663012677 wait complete. 2024-03-18T14:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 Processor -- start 2024-03-18T14:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 -- start start 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19480cfea0 0x7f1948151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19480d1d90 0x7f1948149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f194815bd40 0x7f1948157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f19480d0c40 con 0x7f19480cfea0 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19480d0ac0 con 0x7f19480d1d90 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194fdbc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f194815c120 con 0x7f194815bd40 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194db58640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19480cfea0 0x7f1948151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194db58640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19480cfea0 0x7f1948151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55548/0 (socket says 172.21.15.6:55548) 2024-03-18T14:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.922+0000 7f194db58640 1 -- 172.21.15.6:0/2653363030 learned_addr learned my addr 172.21.15.6:0/2653363030 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194e359640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f194815bd40 0x7f1948157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194d357640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19480d1d90 0x7f1948149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194db58640 1 -- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f194815bd40 msgr2=0x7f1948157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194db58640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f194815bd40 0x7f1948157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194db58640 1 -- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19480d1d90 msgr2=0x7f1948149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194db58640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19480d1d90 0x7f1948149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194db58640 1 -- 172.21.15.6:0/2653363030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19480b5540 con 0x7f19480cfea0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194d357640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19480d1d90 0x7f1948149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194db58640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19480cfea0 0x7f1948151080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f194002bee0 tx=0x7f19400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f19467fc640 1 -- 172.21.15.6:0/2653363030 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940041070 con 0x7f19480cfea0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f19467fc640 1 -- 172.21.15.6:0/2653363030 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f194003f070 con 0x7f19480cfea0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194806c470 con 0x7f19480cfea0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f19467fc640 1 -- 172.21.15.6:0/2653363030 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940035040 con 0x7f19480cfea0 2024-03-18T14:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.923+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194814a4b0 con 0x7f19480cfea0 2024-03-18T14:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.924+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194809fa90 con 0x7f19480cfea0 2024-03-18T14:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.925+0000 7f19467fc640 1 -- 172.21.15.6:0/2653363030 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1940038020 con 0x7f19480cfea0 2024-03-18T14:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.925+0000 7f19467fc640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f193406bbd0 0x7f193406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.925+0000 7f19467fc640 1 -- 172.21.15.6:0/2653363030 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f19400ac6d0 con 0x7f19480cfea0 2024-03-18T14:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.925+0000 7f194d357640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f193406bbd0 0x7f193406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.926+0000 7f194d357640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f193406bbd0 0x7f193406e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19480cec90 tx=0x7f193c007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:14.929+0000 7f19467fc640 1 -- 172.21.15.6:0/2653363030 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1940031800 con 0x7f19480cfea0 2024-03-18T14:42:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.152+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f194804f620 con 0x7f19480cfea0 2024-03-18T14:42:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.152+0000 7f19467fc640 1 -- 172.21.15.6:0/2653363030 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f194007ae80 con 0x7f19480cfea0 2024-03-18T14:42:15.155 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f193406bbd0 msgr2=0x7f193406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f193406bbd0 0x7f193406e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f19480cec90 tx=0x7f193c007c10 comp rx=0 tx=0).stop 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19480cfea0 msgr2=0x7f1948151080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19480cfea0 0x7f1948151080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f194002bee0 tx=0x7f19400046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 shutdown_connections 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f193406bbd0 0x7f193406e090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f194815bd40 0x7f1948157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19480d1d90 0x7f1948149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 --2- 172.21.15.6:0/2653363030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19480cfea0 0x7f1948151080 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 >> 172.21.15.6:0/2653363030 conn(0x7f194809ae20 msgr2=0x7f194815a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 shutdown_connections 2024-03-18T14:42:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:15.154+0000 7f194fdbc640 1 -- 172.21.15.6:0/2653363030 wait complete. 2024-03-18T14:42:15.167 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:42:15.167 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:42:15.167 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi092.front.sepia.ceph.com 2024-03-18T14:42:15.167 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-18T14:42:15.167 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:15.167 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:42:15.167 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:42:15.168 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:42:15.168 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-18T14:42:15.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:15.168 DEBUG:teuthology.orchestra.run.smithi092:> ip addr 2024-03-18T14:42:15.183 INFO:teuthology.orchestra.run.smithi092.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:15.183 INFO:teuthology.orchestra.run.smithi092.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:15.183 INFO:teuthology.orchestra.run.smithi092.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:15.183 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.183 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:15.183 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b2 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f0 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b3 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f1 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:44 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:45 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: inet 172.21.15.92/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft 42646sec preferred_lft 42646sec 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::ec4:7aff:fe8f:cd45/64 scope link 2024-03-18T14:42:15.184 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether de:26:38:d5:f1:ec brd ff:ff:ff:ff:ff:ff 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::8c01:68ff:fe49:3927/64 scope link 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether de:26:38:d5:f1:ec brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::dc26:38ff:fed5:f1ec/64 scope link tentative 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:15.185 DEBUG:teuthology.orchestra.run.smithi092:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:42:15.248 INFO:teuthology.orchestra.run.smithi092.stdout:1 2024-03-18T14:42:15.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:15.250 DEBUG:teuthology.orchestra.run.smithi092:> route 2024-03-18T14:42:15.306 INFO:teuthology.orchestra.run.smithi092.stdout:Kernel IP routing table 2024-03-18T14:42:15.306 INFO:teuthology.orchestra.run.smithi092.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:42:15.306 INFO:teuthology.orchestra.run.smithi092.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T14:42:15.306 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T14:42:15.306 INFO:teuthology.orchestra.run.smithi092.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:42:15.307 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:15.307 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:15.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T14:42:15.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T14:42:15.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T14:42:15.307 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:15.394 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:15.437 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:15.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:15.443 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list 2024-03-18T14:42:15.458 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-18T14:42:15.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:15.458 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list-id 2024-03-18T14:42:15.513 INFO:teuthology.orchestra.run.smithi092.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-18T14:42:15.513 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:15.514 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:15.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T14:42:15.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-18T14:42:15.514 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:15.602 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:15.629 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:15.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:15.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-18T14:42:15.665 INFO:teuthology.orchestra.run.smithi092.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 86:c1:f7:fd:3f:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::84c1:f7ff:fefd:3ff5/64 scope link tentative 2024-03-18T14:42:15.666 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:15.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-18T14:42:15.733 INFO:teuthology.orchestra.run.smithi092.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:42:15.733 INFO:teuthology.orchestra.run.smithi092.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:42:15.733 INFO:teuthology.orchestra.run.smithi092.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 ::1/128 scope host 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 86:c1:f7:fd:3f:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::84c1:f7ff:fefd:3ff5/64 scope link 2024-03-18T14:42:15.734 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:42:15.735 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-18T14:42:15.735 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:15.735 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:15.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-18T14:42:15.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-18T14:42:15.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-18T14:42:15.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-18T14:42:15.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-18T14:42:15.736 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:15.823 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:15.897 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:15.905 DEBUG:teuthology.orchestra.run.smithi092:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:42:15.905 DEBUG:teuthology.orchestra.run.smithi092:> set -e 2024-03-18T14:42:15.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set brx.1 up 2024-03-18T14:42:15.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link set dev brx.1 master ceph-brx 2024-03-18T14:42:15.905 DEBUG:teuthology.orchestra.run.smithi092:> ') 2024-03-18T14:42:15.953 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:42:15.981 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:15.987 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-18T14:42:15.987 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:15.987 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:16.001 INFO:teuthology.orchestra.run.smithi092.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-18T14:42:16.001 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:42:16.001 DEBUG:teuthology.orchestra.run.smithi092:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:16.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo modprobe fuse 2024-03-18T14:42:16.122 DEBUG:teuthology.orchestra.run.smithi092:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/proc 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/sys 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/dev 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/security 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/shm 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/pts 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/run 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/cgroup 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/pstore 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/bpf 2024-03-18T14:42:16.179 INFO:teuthology.orchestra.run.smithi092.stdout:/ 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/selinux 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/debug 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/mqueue 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/hugepages 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/tracing 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/config 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/run/user/1000 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph 2024-03-18T14:42:16.180 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-4 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-5 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-6 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-7 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T14:42:16.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.181 DEBUG:teuthology.orchestra.run.smithi092:> ls /sys/fs/fuse/connections 2024-03-18T14:42:16.234 INFO:teuthology.orchestra.run.smithi092.stdout:42 2024-03-18T14:42:16.234 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-18T14:42:16.235 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-18T14:42:16.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo modprobe fuse 2024-03-18T14:42:16.303 DEBUG:teuthology.orchestra.run.smithi092:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:42:16.359 INFO:teuthology.orchestra.run.smithi092.stdout:/proc 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/sys 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/dev 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/security 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/shm 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/pts 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/run 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/cgroup 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/pstore 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/bpf 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/ 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/selinux 2024-03-18T14:42:16.360 INFO:teuthology.orchestra.run.smithi092.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/debug 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/mqueue 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/hugepages 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/tracing 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/fs/fuse/connections 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/sys/kernel/config 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/run/user/1000 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-4 2024-03-18T14:42:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-5 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-6 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/var/lib/ceph/osd/ceph-7 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T14:42:16.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.362 DEBUG:teuthology.orchestra.run.smithi092:> ls /sys/fs/fuse/connections 2024-03-18T14:42:16.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi092.stderr:2024-03-18T14:42:16.360+0000 7fa921ab7480 -1 init, newargv = 0x563d55edcc40 newargc=15 2024-03-18T14:42:16.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi092.stderr:ceph-fuse[44621]: starting ceph client 2024-03-18T14:42:16.368 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi092.stderr:ceph-fuse[44621]: starting fuse 2024-03-18T14:42:16.378 INFO:teuthology.orchestra.run.smithi092.stdout:42 2024-03-18T14:42:16.378 INFO:teuthology.orchestra.run.smithi092.stdout:44 2024-03-18T14:42:16.378 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-18T14:42:16.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:42:16.378 DEBUG:teuthology.orchestra.run.smithi092:> import glob 2024-03-18T14:42:16.378 DEBUG:teuthology.orchestra.run.smithi092:> import re 2024-03-18T14:42:16.378 DEBUG:teuthology.orchestra.run.smithi092:> import os 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> import subprocess 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> def _find_admin_socket(client_name): 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> files = glob.glob(asok_path) 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> # Given a non-glob path, it better be there 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> if "*" not in asok_path: 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> assert(len(files) == 1) 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> return files[0] 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> for f in files: 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:42:16.379 DEBUG:teuthology.orchestra.run.smithi092:> contents = proc_f.read() 2024-03-18T14:42:16.380 DEBUG:teuthology.orchestra.run.smithi092:> if mountpoint in contents: 2024-03-18T14:42:16.380 DEBUG:teuthology.orchestra.run.smithi092:> return f 2024-03-18T14:42:16.380 DEBUG:teuthology.orchestra.run.smithi092:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:42:16.380 DEBUG:teuthology.orchestra.run.smithi092:> 2024-03-18T14:42:16.380 DEBUG:teuthology.orchestra.run.smithi092:> print(_find_admin_socket("client.3")) 2024-03-18T14:42:16.380 DEBUG:teuthology.orchestra.run.smithi092:> ' 2024-03-18T14:42:16.500 INFO:teuthology.orchestra.run.smithi092.stdout:/var/run/ceph/ceph-client.3.44621.asok 2024-03-18T14:42:16.504 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-18T14:42:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:42:16.511 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44621.asok 2024-03-18T14:42:16.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44621.asok status 2024-03-18T14:42:16.650 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "metadata": { 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "entity_id": "3", 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "hostname": "smithi092", 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "pid": "44621", 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "root": "/" 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "dentry_count": 0, 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "dentry_pinned_count": 0, 2024-03-18T14:42:16.651 INFO:teuthology.orchestra.run.smithi092.stdout: "id": 4685, 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "inst": { 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "name": { 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "client", 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "num": 4685 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": { 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "192.168.0.2:0", 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 1810013851 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": { 2024-03-18T14:42:16.652 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "192.168.0.2:0", 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 1810013851 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "inst_str": "client.4685 192.168.0.2:0/1810013851", 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "addr_str": "192.168.0.2:0/1810013851", 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "inode_count": 1, 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_epoch": 8, 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_epoch": 26, 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "blocklisted": false, 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout: "fs_name": "cephfs" 2024-03-18T14:42:16.653 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-18T14:42:16.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.659 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:42:16.659 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:16.681 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:42:16.681 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:16.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:16.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.752 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:42:16.752 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:16.809 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:42:16.809 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:16.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:16.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.878 DEBUG:teuthology.orchestra.run.smithi092:> stat --file-system '--printf=%T 2024-03-18T14:42:16.878 DEBUG:teuthology.orchestra.run.smithi092:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:16.894 INFO:teuthology.orchestra.run.smithi092.stdout:fuseblk 2024-03-18T14:42:16.894 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:16.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:16.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:16.960 DEBUG:teuthology.orchestra.run.smithi092:> stat --file-system '--printf=%T 2024-03-18T14:42:16.961 DEBUG:teuthology.orchestra.run.smithi092:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:17.015 INFO:teuthology.orchestra.run.smithi092.stdout:fuseblk 2024-03-18T14:42:17.015 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:17.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:17.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:17.082 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-18T14:42:17.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:17.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.207+0000 7f37c7b53640 1 Processor -- start 2024-03-18T14:42:17.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.207+0000 7f37c7b53640 1 -- start start 2024-03-18T14:42:17.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.208+0000 7f37c7b53640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c0162380 0x7f37c00d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.208+0000 7f37c7b53640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c00b9660 con 0x7f37c00ced00 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.208+0000 7f37c7b53640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c0162810 con 0x7f37c015e020 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.208+0000 7f37c7b53640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f37c00d2270 con 0x7f37c0162380 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.208+0000 7f37c50ee640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f37c00ced00 0x7f37c00d3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54976/0 (socket says 172.21.15.6:54976) 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.208+0000 7f37c50ee640 1 -- 172.21.15.6:0/1742159597 learned_addr learned my addr 172.21.15.6:0/1742159597 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.208+0000 7f37c60f0640 1 --2- 172.21.15.6:0/1742159597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c0162380 0x7f37c00d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.209+0000 7f37c48ed640 1 -- 172.21.15.6:0/1742159597 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1993884637 0 0) 0x7f37c00b9660 con 0x7f37c00ced00 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.209+0000 7f37c48ed640 1 -- 172.21.15.6:0/1742159597 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37ac003520 con 0x7f37c00ced00 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.209+0000 7f37c48ed640 1 -- 172.21.15.6:0/1742159597 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335146309 0 0) 0x7f37c0162810 con 0x7f37c015e020 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.209+0000 7f37c48ed640 1 -- 172.21.15.6:0/1742159597 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c00b9660 con 0x7f37c015e020 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.209+0000 7f37c60f0640 1 -- 172.21.15.6:0/1742159597 >> v1:172.21.15.92:6789/0 conn(0x7f37c015e020 legacy=0x7f37c00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.209+0000 7f37c60f0640 1 -- 172.21.15.6:0/1742159597 >> v1:172.21.15.6:6789/0 conn(0x7f37c00ced00 legacy=0x7f37c00d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.209+0000 7f37c60f0640 1 -- 172.21.15.6:0/1742159597 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c00ce390 con 0x7f37c0162380 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c60f0640 1 --2- 172.21.15.6:0/1742159597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c0162380 0x7f37c00d3c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009dc0 tx=0x7f37bc02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4497901362b850e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c48ed640 1 -- 172.21.15.6:0/1742159597 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc039070 con 0x7f37c0162380 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c48ed640 1 -- 172.21.15.6:0/1742159597 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37bc02cc40 con 0x7f37c0162380 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c48ed640 1 -- 172.21.15.6:0/1742159597 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc034440 con 0x7f37c0162380 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c7b53640 1 -- 172.21.15.6:0/1742159597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c0162380 msgr2=0x7f37c00d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c7b53640 1 --2- 172.21.15.6:0/1742159597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c0162380 0x7f37c00d3c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009dc0 tx=0x7f37bc02be90 comp rx=0 tx=0).stop 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c7b53640 1 -- 172.21.15.6:0/1742159597 shutdown_connections 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c7b53640 1 --2- 172.21.15.6:0/1742159597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c0162380 0x7f37c00d3c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.210+0000 7f37c7b53640 1 -- 172.21.15.6:0/1742159597 >> 172.21.15.6:0/1742159597 conn(0x7f37c0072ce0 msgr2=0x7f37c00730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.211+0000 7f37c7b53640 1 -- 172.21.15.6:0/1742159597 shutdown_connections 2024-03-18T14:42:17.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.211+0000 7f37c7b53640 1 -- 172.21.15.6:0/1742159597 wait complete. 2024-03-18T14:42:17.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 Processor -- start 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 -- start start 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c00ced00 0x7f37c00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37c015e020 0x7f37c00cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37c0162380 0x7f37c015e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f37c00d3920 con 0x7f37c015e020 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f37c00b9be0 con 0x7f37c0162380 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.212+0000 7f37c7b53640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f37c00ce890 con 0x7f37c00ced00 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37c015e020 0x7f37c00cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37c015e020 0x7f37c00cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55558/0 (socket says 172.21.15.6:55558) 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 -- 172.21.15.6:0/2207354922 learned_addr learned my addr 172.21.15.6:0/2207354922 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c58ef640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c00ced00 0x7f37c00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c60f0640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37c0162380 0x7f37c015e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 -- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c00ced00 msgr2=0x7f37c00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c00ced00 0x7f37c00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 -- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37c0162380 msgr2=0x7f37c015e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37c0162380 0x7f37c015e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 -- 172.21.15.6:0/2207354922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c014d8a0 con 0x7f37c015e020 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c50ee640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37c015e020 0x7f37c00cd4f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c500 tx=0x7f37b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37b67fc640 1 -- 172.21.15.6:0/2207354922 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0017070 con 0x7f37c015e020 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37b67fc640 1 -- 172.21.15.6:0/2207354922 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b0005ce0 con 0x7f37c015e020 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c01f5040 con 0x7f37c015e020 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37b67fc640 1 -- 172.21.15.6:0/2207354922 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b00072f0 con 0x7f37c015e020 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.213+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c01f4d60 con 0x7f37c015e020 2024-03-18T14:42:17.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.214+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c008afe0 con 0x7f37c015e020 2024-03-18T14:42:17.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.215+0000 7f37b67fc640 1 -- 172.21.15.6:0/2207354922 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f37b001a070 con 0x7f37c015e020 2024-03-18T14:42:17.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.215+0000 7f37b67fc640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37a406bbd0 0x7f37a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.215+0000 7f37b67fc640 1 -- 172.21.15.6:0/2207354922 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f37b008a080 con 0x7f37c015e020 2024-03-18T14:42:17.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.215+0000 7f37c58ef640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37a406bbd0 0x7f37a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.216+0000 7f37c58ef640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37a406bbd0 0x7f37a406e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f37b80010e0 tx=0x7f37b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.220+0000 7f37b67fc640 1 -- 172.21.15.6:0/2207354922 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f37b00581b0 con 0x7f37c015e020 2024-03-18T14:42:17.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.403+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37c00d1940 con 0x7f37c015e020 2024-03-18T14:42:17.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.404+0000 7f37b67fc640 1 -- 172.21.15.6:0/2207354922 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7f37b005c250 con 0x7f37c015e020 2024-03-18T14:42:17.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:17.407 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37a406bbd0 msgr2=0x7f37a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37a406bbd0 0x7f37a406e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f37b80010e0 tx=0x7f37b8009040 comp rx=0 tx=0).stop 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37c015e020 msgr2=0x7f37c00cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37c015e020 0x7f37c00cd4f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c500 tx=0x7f37b000c9c0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 shutdown_connections 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37a406bbd0 0x7f37a406e090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37c0162380 0x7f37c015e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37c015e020 0x7f37c00cd4f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 --2- 172.21.15.6:0/2207354922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37c00ced00 0x7f37c00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 >> 172.21.15.6:0/2207354922 conn(0x7f37c0072ce0 msgr2=0x7f37c00b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 shutdown_connections 2024-03-18T14:42:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.406+0000 7f37c7b53640 1 -- 172.21.15.6:0/2207354922 wait complete. 2024-03-18T14:42:17.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:17.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:17.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 Processor -- start 2024-03-18T14:42:17.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 -- start start 2024-03-18T14:42:17.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa43415dfd0 0x7fa4340b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340b5740 0x7fa4340b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4340d1e40 0x7fa4340d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa4340b60e0 con 0x7fa4340d1e40 2024-03-18T14:42:17.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa43415e3b0 con 0x7fa43415dfd0 2024-03-18T14:42:17.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.522+0000 7fa438a0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa4340b8470 con 0x7fa4340b5740 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4340d1e40 0x7fa4340d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4340d1e40 0x7fa4340d22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55570/0 (socket says 172.21.15.6:55570) 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 -- 172.21.15.6:0/2176057633 learned_addr learned my addr 172.21.15.6:0/2176057633 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa42bfff640 1 --2- 172.21.15.6:0/2176057633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340b5740 0x7fa4340b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa43259c640 1 --2- 172.21.15.6:0/2176057633 >> v2:172.21.15.92:3300/0 conn(0x7fa43415dfd0 0x7fa4340b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 -- 172.21.15.6:0/2176057633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340b5740 msgr2=0x7fa4340b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 --2- 172.21.15.6:0/2176057633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340b5740 0x7fa4340b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 -- 172.21.15.6:0/2176057633 >> v2:172.21.15.92:3300/0 conn(0x7fa43415dfd0 msgr2=0x7fa4340b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 --2- 172.21.15.6:0/2176057633 >> v2:172.21.15.92:3300/0 conn(0x7fa43415dfd0 0x7fa4340b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.523+0000 7fa432d9d640 1 -- 172.21.15.6:0/2176057633 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4340b8750 con 0x7fa4340d1e40 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa432d9d640 1 --2- 172.21.15.6:0/2176057633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340d1e40 0x7fa4340d22d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa4340d17c0 tx=0x7fa42c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2d15d2cb34af31 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa431d9b640 1 -- 172.21.15.6:0/2176057633 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c02ca70 con 0x7fa4340d1e40 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa431d9b640 1 -- 172.21.15.6:0/2176057633 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42c02cc10 con 0x7fa4340d1e40 2024-03-18T14:42:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa431d9b640 1 -- 172.21.15.6:0/2176057633 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c0343f0 con 0x7fa4340d1e40 2024-03-18T14:42:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 -- 172.21.15.6:0/2176057633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340d1e40 msgr2=0x7fa4340d22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 --2- 172.21.15.6:0/2176057633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340d1e40 0x7fa4340d22d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa4340d17c0 tx=0x7fa42c02bed0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 -- 172.21.15.6:0/2176057633 shutdown_connections 2024-03-18T14:42:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 --2- 172.21.15.6:0/2176057633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340d1e40 0x7fa4340d22d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 --2- 172.21.15.6:0/2176057633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340b5740 0x7fa4340b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 --2- 172.21.15.6:0/2176057633 >> v2:172.21.15.92:3300/0 conn(0x7fa43415dfd0 0x7fa4340b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 -- 172.21.15.6:0/2176057633 >> 172.21.15.6:0/2176057633 conn(0x7fa434072d10 msgr2=0x7fa434073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 -- 172.21.15.6:0/2176057633 shutdown_connections 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.524+0000 7fa438a0b640 1 -- 172.21.15.6:0/2176057633 wait complete. 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 Processor -- start 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 -- start start 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340b5740 0x7fa434160e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340d1e40 0x7fa4340ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43415dfd0 0x7fa4340aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4340b9b90 con 0x7fa4340b5740 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa4340b8d10 con 0x7fa43415dfd0 2024-03-18T14:42:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa438a0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa4340d2810 con 0x7fa4340d1e40 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa43259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340b5740 0x7fa434160e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa43259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340b5740 0x7fa434160e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55576/0 (socket says 172.21.15.6:55576) 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa43259c640 1 -- 172.21.15.6:0/3502782931 learned_addr learned my addr 172.21.15.6:0/3502782931 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.526+0000 7fa42bfff640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340d1e40 0x7fa4340ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa432d9d640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43415dfd0 0x7fa4340aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa43259c640 1 -- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340d1e40 msgr2=0x7fa4340ac690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa43259c640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340d1e40 0x7fa4340ac690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa43259c640 1 -- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43415dfd0 msgr2=0x7fa4340aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa43259c640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43415dfd0 0x7fa4340aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa43259c640 1 -- 172.21.15.6:0/3502782931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4341f73b0 con 0x7fa4340b5740 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa43259c640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340b5740 0x7fa434160e00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa42400e870 tx=0x7fa42400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa42affd640 1 -- 172.21.15.6:0/3502782931 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa424017070 con 0x7fa4340b5740 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa42affd640 1 -- 172.21.15.6:0/3502782931 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42400c940 con 0x7fa4340b5740 2024-03-18T14:42:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4341f7690 con 0x7fa4340b5740 2024-03-18T14:42:17.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa42affd640 1 -- 172.21.15.6:0/3502782931 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42400cd30 con 0x7fa4340b5740 2024-03-18T14:42:17.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.527+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4341f7c50 con 0x7fa4340b5740 2024-03-18T14:42:17.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.528+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4340751b0 con 0x7fa4340b5740 2024-03-18T14:42:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.529+0000 7fa42affd640 1 -- 172.21.15.6:0/3502782931 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa42401a070 con 0x7fa4340b5740 2024-03-18T14:42:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.529+0000 7fa42affd640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bbd0 0x7fa41406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.529+0000 7fa42affd640 1 -- 172.21.15.6:0/3502782931 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa42408b060 con 0x7fa4340b5740 2024-03-18T14:42:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.529+0000 7fa42bfff640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bbd0 0x7fa41406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.530+0000 7fa42bfff640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bbd0 0x7fa41406e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009ac0 tx=0x7fa41c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.533+0000 7fa42affd640 1 -- 172.21.15.6:0/3502782931 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa424059100 con 0x7fa4340b5740 2024-03-18T14:42:17.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.717+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4340b5500 con 0x7fa4340b5740 2024-03-18T14:42:17.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.717+0000 7fa42affd640 1 -- 172.21.15.6:0/3502782931 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fa42405d1a0 con 0x7fa4340b5740 2024-03-18T14:42:17.720 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bbd0 msgr2=0x7fa41406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bbd0 0x7fa41406e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009ac0 tx=0x7fa41c009190 comp rx=0 tx=0).stop 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340b5740 msgr2=0x7fa434160e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340b5740 0x7fa434160e00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa42400e870 tx=0x7fa42400ed30 comp rx=0 tx=0).stop 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 shutdown_connections 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bbd0 0x7fa41406e090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43415dfd0 0x7fa4340aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4340d1e40 0x7fa4340ac690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 --2- 172.21.15.6:0/3502782931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4340b5740 0x7fa434160e00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 >> 172.21.15.6:0/3502782931 conn(0x7fa434072d10 msgr2=0x7fa4340ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:17.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 shutdown_connections 2024-03-18T14:42:17.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.719+0000 7fa438a0b640 1 -- 172.21.15.6:0/3502782931 wait complete. 2024-03-18T14:42:17.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:17.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.834+0000 7f9a96ebd640 1 Processor -- start 2024-03-18T14:42:17.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.835+0000 7f9a96ebd640 1 -- start start 2024-03-18T14:42:17.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.835+0000 7f9a96ebd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a9015dfd0 0x7f9a900b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.835+0000 7f9a96ebd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a9015e3b0 con 0x7f9a900b5740 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.835+0000 7f9a96ebd640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a900b8470 con 0x7f9a900b7e00 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.835+0000 7f9a96ebd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9a900b8750 con 0x7f9a9015dfd0 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.835+0000 7f9a8ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a900b5740 0x7f9a900d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54992/0 (socket says 172.21.15.6:54992) 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.835+0000 7f9a8ffff640 1 -- 172.21.15.6:0/1712155945 learned_addr learned my addr 172.21.15.6:0/1712155945 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a94c59640 1 --2- 172.21.15.6:0/1712155945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a9015dfd0 0x7f9a900b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a8f7fe640 1 -- 172.21.15.6:0/1712155945 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1037075233 0 0) 0x7f9a9015e3b0 con 0x7f9a900b5740 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a8f7fe640 1 -- 172.21.15.6:0/1712155945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a7c003520 con 0x7f9a900b5740 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a8f7fe640 1 -- 172.21.15.6:0/1712155945 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407024333 0 0) 0x7f9a900b8470 con 0x7f9a900b7e00 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a8f7fe640 1 -- 172.21.15.6:0/1712155945 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a9015e3b0 con 0x7f9a900b7e00 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a94c59640 1 -- 172.21.15.6:0/1712155945 >> v1:172.21.15.92:6789/0 conn(0x7f9a900b7e00 legacy=0x7f9a900d2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a94c59640 1 -- 172.21.15.6:0/1712155945 >> v1:172.21.15.6:6789/0 conn(0x7f9a900b5740 legacy=0x7f9a900d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.836+0000 7f9a94c59640 1 -- 172.21.15.6:0/1712155945 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a900b8a30 con 0x7f9a9015dfd0 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a94c59640 1 --2- 172.21.15.6:0/1712155945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a9015dfd0 0x7f9a900b9650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097d0 tx=0x7f9a8802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de5410e1490b104 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a8f7fe640 1 -- 172.21.15.6:0/1712155945 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88035020 con 0x7f9a9015dfd0 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a8f7fe640 1 -- 172.21.15.6:0/1712155945 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a88031890 con 0x7f9a9015dfd0 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a8f7fe640 1 -- 172.21.15.6:0/1712155945 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88031c60 con 0x7f9a9015dfd0 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a96ebd640 1 -- 172.21.15.6:0/1712155945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a9015dfd0 msgr2=0x7f9a900b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/1712155945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a9015dfd0 0x7f9a900b9650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097d0 tx=0x7f9a8802b930 comp rx=0 tx=0).stop 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a96ebd640 1 -- 172.21.15.6:0/1712155945 shutdown_connections 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/1712155945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a9015dfd0 0x7f9a900b9650 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a96ebd640 1 -- 172.21.15.6:0/1712155945 >> 172.21.15.6:0/1712155945 conn(0x7f9a90072d10 msgr2=0x7f9a90073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a96ebd640 1 -- 172.21.15.6:0/1712155945 shutdown_connections 2024-03-18T14:42:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.837+0000 7f9a96ebd640 1 -- 172.21.15.6:0/1712155945 wait complete. 2024-03-18T14:42:17.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.838+0000 7f9a96ebd640 1 Processor -- start 2024-03-18T14:42:17.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a96ebd640 1 -- start start 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a96ebd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a900b5740 0x7f9a900d36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a96ebd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a900b7e00 0x7f9a900d3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a96ebd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9015dfd0 0x7f9a9015e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a96ebd640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a900d2c60 con 0x7f9a900b5740 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a96ebd640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a900b9b90 con 0x7f9a9015dfd0 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a96ebd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9a900d2de0 con 0x7f9a900b7e00 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a94c59640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a900b5740 0x7f9a900d36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a94c59640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a900b5740 0x7f9a900d36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55578/0 (socket says 172.21.15.6:55578) 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.839+0000 7f9a94c59640 1 -- 172.21.15.6:0/2658950341 learned_addr learned my addr 172.21.15.6:0/2658950341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a8ffff640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a900b7e00 0x7f9a900d3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a9545a640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9015dfd0 0x7f9a9015e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a94c59640 1 -- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a900b7e00 msgr2=0x7f9a900d3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a94c59640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a900b7e00 0x7f9a900d3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a94c59640 1 -- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9015dfd0 msgr2=0x7f9a9015e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a94c59640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9015dfd0 0x7f9a9015e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a94c59640 1 -- 172.21.15.6:0/2658950341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a901f7ff0 con 0x7f9a900b5740 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a9545a640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9015dfd0 0x7f9a9015e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a94c59640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a900b5740 0x7f9a900d36d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a88000c00 tx=0x7f9a880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a8d7fa640 1 -- 172.21.15.6:0/2658950341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88035040 con 0x7f9a900b5740 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a8d7fa640 1 -- 172.21.15.6:0/2658950341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a88030d20 con 0x7f9a900b5740 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a900c81a0 con 0x7f9a900b5740 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a8d7fa640 1 -- 172.21.15.6:0/2658950341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a880456e0 con 0x7f9a900b5740 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.840+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a901f7510 con 0x7f9a900b5740 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.841+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a900751b0 con 0x7f9a900b5740 2024-03-18T14:42:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.842+0000 7f9a8d7fa640 1 -- 172.21.15.6:0/2658950341 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9a88038020 con 0x7f9a900b5740 2024-03-18T14:42:17.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.842+0000 7f9a8d7fa640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a7006bc00 0x7f9a7006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:17.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.842+0000 7f9a8d7fa640 1 -- 172.21.15.6:0/2658950341 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9a880ad030 con 0x7f9a900b5740 2024-03-18T14:42:17.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.842+0000 7f9a8ffff640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a7006bc00 0x7f9a7006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:17.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.843+0000 7f9a8ffff640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a7006bc00 0x7f9a7006e0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9a90068560 tx=0x7f9a80001ad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:17.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:17.846+0000 7f9a8d7fa640 1 -- 172.21.15.6:0/2658950341 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9a8802d040 con 0x7f9a900b5740 2024-03-18T14:42:18.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.026+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a900b3ea0 con 0x7f9a900b5740 2024-03-18T14:42:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.026+0000 7f9a8d7fa640 1 -- 172.21.15.6:0/2658950341 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f9a8807b7b0 con 0x7f9a900b5740 2024-03-18T14:42:18.029 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a7006bc00 msgr2=0x7f9a7006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a7006bc00 0x7f9a7006e0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9a90068560 tx=0x7f9a80001ad0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a900b5740 msgr2=0x7f9a900d36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a900b5740 0x7f9a900d36d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a88000c00 tx=0x7f9a880046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 shutdown_connections 2024-03-18T14:42:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a7006bc00 0x7f9a7006e0c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9015dfd0 0x7f9a9015e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a900b7e00 0x7f9a900d3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 --2- 172.21.15.6:0/2658950341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a900b5740 0x7f9a900d36d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 >> 172.21.15.6:0/2658950341 conn(0x7f9a90072d10 msgr2=0x7f9a900b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 shutdown_connections 2024-03-18T14:42:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.028+0000 7f9a96ebd640 1 -- 172.21.15.6:0/2658950341 wait complete. 2024-03-18T14:42:18.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:18.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.144+0000 7f471439e640 1 Processor -- start 2024-03-18T14:42:18.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.144+0000 7f471439e640 1 -- start start 2024-03-18T14:42:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471439e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f470c15cf10 0x7f470c0af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471439e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c15dac0 0x7f470c15df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471439e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c0afef0 con 0x7f470c071ad0 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471439e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f470c15d370 con 0x7f470c15cf10 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471439e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f470c0b01d0 con 0x7f470c15dac0 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471213a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f470c071ad0 0x7f470c071eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54994/0 (socket says 172.21.15.6:54994) 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471213a640 1 -- 172.21.15.6:0/606341482 learned_addr learned my addr 172.21.15.6:0/606341482 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f4711939640 1 --2- 172.21.15.6:0/606341482 >> v2:172.21.15.6:3300/0 conn(0x7f470c15cf10 0x7f470c0af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f471293b640 1 --2- 172.21.15.6:0/606341482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c15dac0 0x7f470c15df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f4711138640 1 -- 172.21.15.6:0/606341482 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373776479 0 0) 0x7f470c0afef0 con 0x7f470c071ad0 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f4711138640 1 -- 172.21.15.6:0/606341482 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f4003520 con 0x7f470c071ad0 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f4711939640 1 -- 172.21.15.6:0/606341482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c15dac0 msgr2=0x7f470c15df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f4711939640 1 --2- 172.21.15.6:0/606341482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c15dac0 0x7f470c15df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f4711939640 1 -- 172.21.15.6:0/606341482 >> v1:172.21.15.6:6789/0 conn(0x7f470c071ad0 legacy=0x7f470c071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.145+0000 7f4711939640 1 -- 172.21.15.6:0/606341482 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c0b04b0 con 0x7f470c15cf10 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.146+0000 7f4711939640 1 --2- 172.21.15.6:0/606341482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0af580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f47000097a0 tx=0x7f470002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4da8ac9fc3432ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.146+0000 7f4711138640 1 -- 172.21.15.6:0/606341482 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700035020 con 0x7f470c15cf10 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.146+0000 7f4711138640 1 -- 172.21.15.6:0/606341482 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4700031830 con 0x7f470c15cf10 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.146+0000 7f4711138640 1 -- 172.21.15.6:0/606341482 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700031c00 con 0x7f470c15cf10 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 -- 172.21.15.6:0/606341482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 msgr2=0x7f470c0af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 --2- 172.21.15.6:0/606341482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0af580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f47000097a0 tx=0x7f470002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 -- 172.21.15.6:0/606341482 shutdown_connections 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 --2- 172.21.15.6:0/606341482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c15dac0 0x7f470c15df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 --2- 172.21.15.6:0/606341482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0af580 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 -- 172.21.15.6:0/606341482 >> 172.21.15.6:0/606341482 conn(0x7f470c06b1c0 msgr2=0x7f470c06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:18.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 -- 172.21.15.6:0/606341482 shutdown_connections 2024-03-18T14:42:18.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.147+0000 7f471439e640 1 -- 172.21.15.6:0/606341482 wait complete. 2024-03-18T14:42:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.148+0000 7f471439e640 1 Processor -- start 2024-03-18T14:42:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.148+0000 7f471439e640 1 -- start start 2024-03-18T14:42:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471439e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c071ad0 0x7f470c13af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471439e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471439e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f470c15dac0 0x7f470c0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471439e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f470c159be0 con 0x7f470c15cf10 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471439e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f470c159a60 con 0x7f470c15dac0 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471439e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f470c0afac0 con 0x7f470c071ad0 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0ae5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55598/0 (socket says 172.21.15.6:55598) 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 -- 172.21.15.6:0/2290025115 learned_addr learned my addr 172.21.15.6:0/2290025115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471213a640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c071ad0 0x7f470c13af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471293b640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f470c15dac0 0x7f470c0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 -- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c071ad0 msgr2=0x7f470c13af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c071ad0 0x7f470c13af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 -- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f470c15dac0 msgr2=0x7f470c0c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f470c15dac0 0x7f470c0c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 -- 172.21.15.6:0/2290025115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c13b8c0 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f471293b640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f470c15dac0 0x7f470c0c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.149+0000 7f4711939640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0ae5c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4700009890 tx=0x7f47000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.150+0000 7f470affd640 1 -- 172.21.15.6:0/2290025115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700035040 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.150+0000 7f470affd640 1 -- 172.21.15.6:0/2290025115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470003b920 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.150+0000 7f470affd640 1 -- 172.21.15.6:0/2290025115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470003bd10 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.150+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c0ad520 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.150+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c0adae0 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.151+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c0751b0 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.151+0000 7f470affd640 1 -- 172.21.15.6:0/2290025115 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4700038020 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.151+0000 7f470affd640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f46f806bbd0 0x7f46f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.151+0000 7f470affd640 1 -- 172.21.15.6:0/2290025115 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f47000acf70 con 0x7f470c15cf10 2024-03-18T14:42:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.151+0000 7f471213a640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f46f806bbd0 0x7f46f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.152+0000 7f471213a640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f46f806bbd0 0x7f46f806e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4704005880 tx=0x7f4704009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.156+0000 7f470affd640 1 -- 172.21.15.6:0/2290025115 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f470007b660 con 0x7f470c15cf10 2024-03-18T14:42:18.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.338+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f470c15aaa0 con 0x7f470c15cf10 2024-03-18T14:42:18.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.339+0000 7f470affd640 1 -- 172.21.15.6:0/2290025115 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f470007af60 con 0x7f470c15cf10 2024-03-18T14:42:18.341 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:18.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f46f806bbd0 msgr2=0x7f46f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f46f806bbd0 0x7f46f806e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4704005880 tx=0x7f4704009040 comp rx=0 tx=0).stop 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 msgr2=0x7f470c0ae5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0ae5c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4700009890 tx=0x7f47000046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 shutdown_connections 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f46f806bbd0 0x7f46f806e090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f470c15dac0 0x7f470c0c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.340+0000 7f471439e640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f470c15cf10 0x7f470c0ae5c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.341+0000 7f471439e640 1 --2- 172.21.15.6:0/2290025115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f470c071ad0 0x7f470c13af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.341+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 >> 172.21.15.6:0/2290025115 conn(0x7f470c06b1c0 msgr2=0x7f470c15e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.341+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 shutdown_connections 2024-03-18T14:42:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.341+0000 7f471439e640 1 -- 172.21.15.6:0/2290025115 wait complete. 2024-03-18T14:42:18.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:18.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.456+0000 7f3f6740b640 1 Processor -- start 2024-03-18T14:42:18.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.456+0000 7f3f6740b640 1 -- start start 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f6740b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f6015dfd0 0x7f3f600b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f6740b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6015e3b0 con 0x7f3f600b5740 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f6740b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f600b8470 con 0x7f3f600b7e00 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f6740b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3f600b8750 con 0x7f3f6015dfd0 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f649a6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f600b5740 0x7f3f600d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55000/0 (socket says 172.21.15.6:55000) 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f649a6640 1 -- 172.21.15.6:0/83226949 learned_addr learned my addr 172.21.15.6:0/83226949 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f651a7640 1 --2- 172.21.15.6:0/83226949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f6015dfd0 0x7f3f600b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f57fff640 1 -- 172.21.15.6:0/83226949 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226501261 0 0) 0x7f3f6015e3b0 con 0x7f3f600b5740 2024-03-18T14:42:18.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f57fff640 1 -- 172.21.15.6:0/83226949 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f4c003520 con 0x7f3f600b5740 2024-03-18T14:42:18.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f57fff640 1 -- 172.21.15.6:0/83226949 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087022699 0 0) 0x7f3f600b8470 con 0x7f3f600b7e00 2024-03-18T14:42:18.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f57fff640 1 -- 172.21.15.6:0/83226949 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6015e3b0 con 0x7f3f600b7e00 2024-03-18T14:42:18.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f651a7640 1 -- 172.21.15.6:0/83226949 >> v1:172.21.15.92:6789/0 conn(0x7f3f600b7e00 legacy=0x7f3f600d2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f651a7640 1 -- 172.21.15.6:0/83226949 >> v1:172.21.15.6:6789/0 conn(0x7f3f600b5740 legacy=0x7f3f600d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.457+0000 7f3f651a7640 1 -- 172.21.15.6:0/83226949 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f600b8a30 con 0x7f3f6015dfd0 2024-03-18T14:42:18.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.458+0000 7f3f651a7640 1 --2- 172.21.15.6:0/83226949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f6015dfd0 0x7f3f600b9650 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0097d0 tx=0x7f3f5c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2a0c212e5ca96e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.458+0000 7f3f57fff640 1 -- 172.21.15.6:0/83226949 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c035020 con 0x7f3f6015dfd0 2024-03-18T14:42:18.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.458+0000 7f3f57fff640 1 -- 172.21.15.6:0/83226949 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f5c031890 con 0x7f3f6015dfd0 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.458+0000 7f3f57fff640 1 -- 172.21.15.6:0/83226949 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c031c40 con 0x7f3f6015dfd0 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.459+0000 7f3f6740b640 1 -- 172.21.15.6:0/83226949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f6015dfd0 msgr2=0x7f3f600b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.459+0000 7f3f6740b640 1 --2- 172.21.15.6:0/83226949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f6015dfd0 0x7f3f600b9650 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0097d0 tx=0x7f3f5c02b930 comp rx=0 tx=0).stop 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.459+0000 7f3f6740b640 1 -- 172.21.15.6:0/83226949 shutdown_connections 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.459+0000 7f3f6740b640 1 --2- 172.21.15.6:0/83226949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f6015dfd0 0x7f3f600b9650 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.459+0000 7f3f6740b640 1 -- 172.21.15.6:0/83226949 >> 172.21.15.6:0/83226949 conn(0x7f3f60072d10 msgr2=0x7f3f60073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.459+0000 7f3f6740b640 1 -- 172.21.15.6:0/83226949 shutdown_connections 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.459+0000 7f3f6740b640 1 -- 172.21.15.6:0/83226949 wait complete. 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.460+0000 7f3f6740b640 1 Processor -- start 2024-03-18T14:42:18.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.460+0000 7f3f6740b640 1 -- start start 2024-03-18T14:42:18.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f6740b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f600b5740 0x7f3f601e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f6740b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f600b7e00 0x7f3f6014d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f6740b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f6015dfd0 0x7f3f600b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f6740b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3f600d2c60 con 0x7f3f6015dfd0 2024-03-18T14:42:18.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f6740b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3f600b9b90 con 0x7f3f600b5740 2024-03-18T14:42:18.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f6740b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3f600d2de0 con 0x7f3f600b7e00 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f659a8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f6015dfd0 0x7f3f600b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f600b5740 0x7f3f601e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f600b5740 0x7f3f601e8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:32824/0 (socket says 172.21.15.6:32824) 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f659a8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f6015dfd0 0x7f3f600b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55604/0 (socket says 172.21.15.6:55604) 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f649a6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f600b7e00 0x7f3f6014d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 -- 172.21.15.6:0/3943699087 learned_addr learned my addr 172.21.15.6:0/3943699087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 -- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f600b7e00 msgr2=0x7f3f6014d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f600b7e00 0x7f3f6014d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 -- 172.21.15.6:0/3943699087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f6015dfd0 msgr2=0x7f3f600b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f6015dfd0 0x7f3f600b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 -- 172.21.15.6:0/3943699087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6014d630 con 0x7f3f600b5740 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f651a7640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f600b5740 0x7f3f601e8fa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c02b970 tx=0x7f3f5c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f55ffb640 1 -- 172.21.15.6:0/3943699087 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c035040 con 0x7f3f600b5740 2024-03-18T14:42:18.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.461+0000 7f3f55ffb640 1 -- 172.21.15.6:0/3943699087 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f5c030d20 con 0x7f3f600b5740 2024-03-18T14:42:18.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.462+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6015f050 con 0x7f3f600b5740 2024-03-18T14:42:18.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.462+0000 7f3f55ffb640 1 -- 172.21.15.6:0/3943699087 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c0456a0 con 0x7f3f600b5740 2024-03-18T14:42:18.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.462+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f6015f5d0 con 0x7f3f600b5740 2024-03-18T14:42:18.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.463+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f600751b0 con 0x7f3f600b5740 2024-03-18T14:42:18.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.463+0000 7f3f55ffb640 1 -- 172.21.15.6:0/3943699087 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3f5c038020 con 0x7f3f600b5740 2024-03-18T14:42:18.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.464+0000 7f3f55ffb640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f4406bbd0 0x7f3f4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.464+0000 7f3f55ffb640 1 -- 172.21.15.6:0/3943699087 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3f5c0ad100 con 0x7f3f600b5740 2024-03-18T14:42:18.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.464+0000 7f3f649a6640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f4406bbd0 0x7f3f4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.464+0000 7f3f649a6640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f4406bbd0 0x7f3f4406e090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f600d2340 tx=0x7f3f50007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.472+0000 7f3f55ffb640 1 -- 172.21.15.6:0/3943699087 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3f6014d630 con 0x7f3f600b5740 2024-03-18T14:42:18.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.653+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3f600d3fa0 con 0x7f3f600b5740 2024-03-18T14:42:18.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.653+0000 7f3f55ffb640 1 -- 172.21.15.6:0/3943699087 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f3f5c02d040 con 0x7f3f600b5740 2024-03-18T14:42:18.656 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f4406bbd0 msgr2=0x7f3f4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f4406bbd0 0x7f3f4406e090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f600d2340 tx=0x7f3f50007d50 comp rx=0 tx=0).stop 2024-03-18T14:42:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f600b5740 msgr2=0x7f3f601e8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f600b5740 0x7f3f601e8fa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c02b970 tx=0x7f3f5c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 shutdown_connections 2024-03-18T14:42:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f4406bbd0 0x7f3f4406e090 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f6015dfd0 0x7f3f600b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f600b7e00 0x7f3f6014d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 --2- 172.21.15.6:0/3943699087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f600b5740 0x7f3f601e8fa0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 >> 172.21.15.6:0/3943699087 conn(0x7f3f60072d10 msgr2=0x7f3f600b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 shutdown_connections 2024-03-18T14:42:18.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.655+0000 7f3f6740b640 1 -- 172.21.15.6:0/3943699087 wait complete. 2024-03-18T14:42:18.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.767+0000 7f038b11b640 1 Processor -- start 2024-03-18T14:42:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.767+0000 7f038b11b640 1 -- start start 2024-03-18T14:42:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f038b11b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840b17b0 0x7f03840b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f038b11b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f03840d0260 0x7f03840d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f038b11b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03840cfd80 con 0x7f038406cce0 2024-03-18T14:42:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f038b11b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f03840b2df0 con 0x7f03840d0260 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f038b11b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f03840b2270 con 0x7f03840b17b0 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f0383fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840b17b0 0x7f03840b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f03896b8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f03840d0260 0x7f03840d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f0383fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840b17b0 0x7f03840b1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60346/0 (socket says 172.21.15.6:60346) 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f0383fff640 1 -- 172.21.15.6:0/2262010490 learned_addr learned my addr 172.21.15.6:0/2262010490 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f0388eb7640 1 --1- 172.21.15.6:0/2262010490 >> v1:172.21.15.92:6789/0 conn(0x7f038406cce0 0x7f038406d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37762/0 (socket says 172.21.15.6:37762) 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.768+0000 7f0383fff640 1 -- 172.21.15.6:0/2262010490 >> v2:172.21.15.92:3300/0 conn(0x7f03840d0260 msgr2=0x7f03840d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.769+0000 7f0383fff640 1 --2- 172.21.15.6:0/2262010490 >> v2:172.21.15.92:3300/0 conn(0x7f03840d0260 0x7f03840d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.769+0000 7f0383fff640 1 -- 172.21.15.6:0/2262010490 >> v1:172.21.15.92:6789/0 conn(0x7f038406cce0 legacy=0x7f038406d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.769+0000 7f03837fe640 1 -- 172.21.15.6:0/2262010490 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883268762 0 0) 0x7f03840cfd80 con 0x7f038406cce0 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.769+0000 7f0383fff640 1 -- 172.21.15.6:0/2262010490 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03840b2550 con 0x7f03840b17b0 2024-03-18T14:42:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.769+0000 7f03896b8640 1 --2- 172.21.15.6:0/2262010490 >> v2:172.21.15.92:3300/0 conn(0x7f03840d0260 0x7f03840d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.769+0000 7f0383fff640 1 --2- 172.21.15.6:0/2262010490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840b17b0 0x7f03840b1c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0374009e50 tx=0x7f037402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3f902dcb27554ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.769+0000 7f03837fe640 1 -- 172.21.15.6:0/2262010490 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037402ca70 con 0x7f03840b17b0 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f03837fe640 1 -- 172.21.15.6:0/2262010490 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037402cc10 con 0x7f03840b17b0 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f03837fe640 1 -- 172.21.15.6:0/2262010490 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03740343f0 con 0x7f03840b17b0 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 -- 172.21.15.6:0/2262010490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840b17b0 msgr2=0x7f03840b1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 --2- 172.21.15.6:0/2262010490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840b17b0 0x7f03840b1c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0374009e50 tx=0x7f037402bed0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 -- 172.21.15.6:0/2262010490 shutdown_connections 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 --2- 172.21.15.6:0/2262010490 >> v2:172.21.15.92:3300/0 conn(0x7f03840d0260 0x7f03840d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 --2- 172.21.15.6:0/2262010490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840b17b0 0x7f03840b1c10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 -- 172.21.15.6:0/2262010490 >> 172.21.15.6:0/2262010490 conn(0x7f038408b130 msgr2=0x7f038408b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 -- 172.21.15.6:0/2262010490 shutdown_connections 2024-03-18T14:42:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.770+0000 7f038b11b640 1 -- 172.21.15.6:0/2262010490 wait complete. 2024-03-18T14:42:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.771+0000 7f038b11b640 1 Processor -- start 2024-03-18T14:42:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.771+0000 7f038b11b640 1 -- start start 2024-03-18T14:42:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f038b11b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f038406cce0 0x7f038414afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f038b11b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03840b17b0 0x7f03840b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f038b11b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840d0260 0x7f03840b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f038b11b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f038415e330 con 0x7f03840b17b0 2024-03-18T14:42:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f038b11b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f03840b2b10 con 0x7f038406cce0 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f038b11b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f038406c690 con 0x7f03840d0260 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f0383fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03840b17b0 0x7f03840b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f0383fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03840b17b0 0x7f03840b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55614/0 (socket says 172.21.15.6:55614) 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f0388eb7640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f038406cce0 0x7f038414afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f03896b8640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840d0260 0x7f03840b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f0383fff640 1 -- 172.21.15.6:0/1703677037 learned_addr learned my addr 172.21.15.6:0/1703677037 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f03896b8640 1 -- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f038406cce0 msgr2=0x7f038414afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f03896b8640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f038406cce0 0x7f038414afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f03896b8640 1 -- 172.21.15.6:0/1703677037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03840b17b0 msgr2=0x7f03840b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.772+0000 7f03896b8640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03840b17b0 0x7f03840b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.773+0000 7f03896b8640 1 -- 172.21.15.6:0/1703677037 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038414b520 con 0x7f03840d0260 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.773+0000 7f03896b8640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840d0260 0x7f03840b6ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f036c00e800 tx=0x7f036c00ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.773+0000 7f03817fa640 1 -- 172.21.15.6:0/1703677037 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c017070 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.773+0000 7f03817fa640 1 -- 172.21.15.6:0/1703677037 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f036c004020 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.773+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03840afd80 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.773+0000 7f03817fa640 1 -- 172.21.15.6:0/1703677037 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c0043f0 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.773+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03840b0340 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.774+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f038409fa80 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.774+0000 7f03817fa640 1 -- 172.21.15.6:0/1703677037 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f036c00c940 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.775+0000 7f03817fa640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f036406bbd0 0x7f036406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.775+0000 7f03817fa640 1 -- 172.21.15.6:0/1703677037 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f036c08aec0 con 0x7f03840d0260 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.775+0000 7f0388eb7640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f036406bbd0 0x7f036406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.775+0000 7f0388eb7640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f036406bbd0 0x7f036406e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f037c000f60 tx=0x7f037c007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:18.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.779+0000 7f03817fa640 1 -- 172.21.15.6:0/1703677037 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f036c058f50 con 0x7f03840d0260 2024-03-18T14:42:18.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.962+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f038404f620 con 0x7f03840d0260 2024-03-18T14:42:18.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.963+0000 7f03817fa640 1 -- 172.21.15.6:0/1703677037 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7f036c05cff0 con 0x7f03840d0260 2024-03-18T14:42:18.966 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:18.966 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f036406bbd0 msgr2=0x7f036406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f036406bbd0 0x7f036406e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f037c000f60 tx=0x7f037c007d50 comp rx=0 tx=0).stop 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840d0260 msgr2=0x7f03840b6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840d0260 0x7f03840b6ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f036c00e800 tx=0x7f036c00ecc0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 shutdown_connections 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f036406bbd0 0x7f036406e090 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03840d0260 0x7f03840b6ff0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03840b17b0 0x7f03840b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 --2- 172.21.15.6:0/1703677037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f038406cce0 0x7f038414afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 >> 172.21.15.6:0/1703677037 conn(0x7f038408b130 msgr2=0x7f038406d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 shutdown_connections 2024-03-18T14:42:18.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:18.965+0000 7f038b11b640 1 -- 172.21.15.6:0/1703677037 wait complete. 2024-03-18T14:42:18.978 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:18.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:19.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.081+0000 7fdc0711b640 1 Processor -- start 2024-03-18T14:42:19.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.081+0000 7fdc0711b640 1 -- start start 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdc0711b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc001443c0 0x7fdc000b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdc0711b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc0015f6c0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdc0711b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc0015e900 con 0x7fdc00143ed0 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdc0711b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fdc000cf7d0 con 0x7fdc001443c0 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdbfffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc001443c0 0x7fdc000b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdc056b8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdc000b0cd0 0x7fdc000b3870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55010/0 (socket says 172.21.15.6:55010) 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdc056b8640 1 -- 172.21.15.6:0/2544116188 learned_addr learned my addr 172.21.15.6:0/2544116188 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdbfffff640 1 -- 172.21.15.6:0/2544116188 >> v1:172.21.15.92:6789/0 conn(0x7fdc00143ed0 legacy=0x7fdc000cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdbfffff640 1 -- 172.21.15.6:0/2544116188 >> v1:172.21.15.6:6789/0 conn(0x7fdc000b0cd0 legacy=0x7fdc000b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdbff7fe640 1 -- 172.21.15.6:0/2544116188 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3787995081 0 0) 0x7fdc0015e900 con 0x7fdc00143ed0 2024-03-18T14:42:19.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.082+0000 7fdbfffff640 1 -- 172.21.15.6:0/2544116188 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0015ebe0 con 0x7fdc001443c0 2024-03-18T14:42:19.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.083+0000 7fdbfffff640 1 --2- 172.21.15.6:0/2544116188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc001443c0 0x7fdc000b0790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0097a0 tx=0x7fdbec02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2410ed9516b6200 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.083+0000 7fdbff7fe640 1 -- 172.21.15.6:0/2544116188 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec037070 con 0x7fdc001443c0 2024-03-18T14:42:19.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.083+0000 7fdbff7fe640 1 -- 172.21.15.6:0/2544116188 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbec030710 con 0x7fdc001443c0 2024-03-18T14:42:19.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.083+0000 7fdbff7fe640 1 -- 172.21.15.6:0/2544116188 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec030ae0 con 0x7fdc001443c0 2024-03-18T14:42:19.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.083+0000 7fdc0711b640 1 -- 172.21.15.6:0/2544116188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc001443c0 msgr2=0x7fdc000b0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.083+0000 7fdc0711b640 1 --2- 172.21.15.6:0/2544116188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc001443c0 0x7fdc000b0790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0097a0 tx=0x7fdbec02b790 comp rx=0 tx=0).stop 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.084+0000 7fdc0711b640 1 -- 172.21.15.6:0/2544116188 shutdown_connections 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.084+0000 7fdc0711b640 1 --2- 172.21.15.6:0/2544116188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc001443c0 0x7fdc000b0790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.084+0000 7fdc0711b640 1 -- 172.21.15.6:0/2544116188 >> 172.21.15.6:0/2544116188 conn(0x7fdc0006b1c0 msgr2=0x7fdc0006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.084+0000 7fdc0711b640 1 -- 172.21.15.6:0/2544116188 shutdown_connections 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.084+0000 7fdc0711b640 1 -- 172.21.15.6:0/2544116188 wait complete. 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.085+0000 7fdc0711b640 1 Processor -- start 2024-03-18T14:42:19.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.085+0000 7fdc0711b640 1 -- start start 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.085+0000 7fdc0711b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc000b0cd0 0x7fdc001e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.085+0000 7fdc0711b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc00143ed0 0x7fdc00159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.085+0000 7fdc0711b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc001443c0 0x7fdc0013b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc0711b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0015f230 con 0x7fdc000b0cd0 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc0711b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc000cf590 con 0x7fdc001443c0 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc0711b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdc000b3f80 con 0x7fdc00143ed0 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc000b0cd0 0x7fdc001e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc000b0cd0 0x7fdc001e6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55632/0 (socket says 172.21.15.6:55632) 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 -- 172.21.15.6:0/766362217 learned_addr learned my addr 172.21.15.6:0/766362217 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc056b8640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc001443c0 0x7fdc0013b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdbfffff640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc00143ed0 0x7fdc00159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 -- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc00143ed0 msgr2=0x7fdc00159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc00143ed0 0x7fdc00159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 -- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc001443c0 msgr2=0x7fdc0013b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc001443c0 0x7fdc0013b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 -- 172.21.15.6:0/766362217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0015a310 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc04eb7640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc000b0cd0 0x7fdc001e6a00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c4e0 tx=0x7fdbf400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdbfd7fa640 1 -- 172.21.15.6:0/766362217 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4017070 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdbfd7fa640 1 -- 172.21.15.6:0/766362217 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf40035f0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0015a5f0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdbfd7fa640 1 -- 172.21.15.6:0/766362217 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40039e0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.086+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0013c3a0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.087+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc0013bed0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.088+0000 7fdbfd7fa640 1 -- 172.21.15.6:0/766362217 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdbf401a070 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.088+0000 7fdbfd7fa640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdbe406bbd0 0x7fdbe406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.088+0000 7fdbfd7fa640 1 -- 172.21.15.6:0/766362217 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdbf408a0a0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.088+0000 7fdbfffff640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdbe406bbd0 0x7fdbe406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.089+0000 7fdbfffff640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdbe406bbd0 0x7fdbe406e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdbec034000 tx=0x7fdbec02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.092+0000 7fdbfd7fa640 1 -- 172.21.15.6:0/766362217 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdbf40581d0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.274+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdc000751b0 con 0x7fdc000b0cd0 2024-03-18T14:42:19.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.274+0000 7fdbfd7fa640 1 -- 172.21.15.6:0/766362217 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fdbf405c270 con 0x7fdc000b0cd0 2024-03-18T14:42:19.277 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdbe406bbd0 msgr2=0x7fdbe406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdbe406bbd0 0x7fdbe406e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdbec034000 tx=0x7fdbec02f000 comp rx=0 tx=0).stop 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc000b0cd0 msgr2=0x7fdc001e6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc000b0cd0 0x7fdc001e6a00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c4e0 tx=0x7fdbf400c9a0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 shutdown_connections 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdbe406bbd0 0x7fdbe406e090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc001443c0 0x7fdc0013b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc00143ed0 0x7fdc00159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 --2- 172.21.15.6:0/766362217 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc000b0cd0 0x7fdc001e6a00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 >> 172.21.15.6:0/766362217 conn(0x7fdc0006b1c0 msgr2=0x7fdc0015e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:19.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 shutdown_connections 2024-03-18T14:42:19.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.276+0000 7fdc0711b640 1 -- 172.21.15.6:0/766362217 wait complete. 2024-03-18T14:42:19.289 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.391+0000 7f12ee2f8640 1 Processor -- start 2024-03-18T14:42:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.391+0000 7f12ee2f8640 1 -- start start 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ee2f8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12e80d13a0 0x7f12e8160310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ee2f8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e8145430 con 0x7f12e8145050 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ee2f8640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e80d0270 con 0x7f12e80b2cf0 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ee2f8640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f12e80d1830 con 0x7f12e80d13a0 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ec895640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12e80d13a0 0x7f12e8160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12e7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12e8145050 0x7f12e80b25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55022/0 (socket says 172.21.15.6:55022) 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12e7fff640 1 -- 172.21.15.6:0/1169526959 learned_addr learned my addr 172.21.15.6:0/1169526959 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ec895640 1 -- 172.21.15.6:0/1169526959 >> v1:172.21.15.92:6789/0 conn(0x7f12e80b2cf0 legacy=0x7f12e815c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ec895640 1 -- 172.21.15.6:0/1169526959 >> v1:172.21.15.6:6789/0 conn(0x7f12e8145050 legacy=0x7f12e80b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12ec895640 1 -- 172.21.15.6:0/1169526959 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e80d0550 con 0x7f12e80d13a0 2024-03-18T14:42:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.392+0000 7f12e6ffd640 1 -- 172.21.15.6:0/1169526959 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888733030 0 0) 0x7f12e8145430 con 0x7f12e8145050 2024-03-18T14:42:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.393+0000 7f12ec895640 1 --2- 172.21.15.6:0/1169526959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e80d13a0 0x7f12e8160310 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009d90 tx=0x7f12e002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706346df49f896a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.393+0000 7f12e6ffd640 1 -- 172.21.15.6:0/1169526959 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0039070 con 0x7f12e80d13a0 2024-03-18T14:42:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.393+0000 7f12e6ffd640 1 -- 172.21.15.6:0/1169526959 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e002cd00 con 0x7f12e80d13a0 2024-03-18T14:42:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.393+0000 7f12e6ffd640 1 -- 172.21.15.6:0/1169526959 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0034660 con 0x7f12e80d13a0 2024-03-18T14:42:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.393+0000 7f12ee2f8640 1 -- 172.21.15.6:0/1169526959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e80d13a0 msgr2=0x7f12e8160310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.393+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/1169526959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e80d13a0 0x7f12e8160310 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009d90 tx=0x7f12e002bf50 comp rx=0 tx=0).stop 2024-03-18T14:42:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.394+0000 7f12ee2f8640 1 -- 172.21.15.6:0/1169526959 shutdown_connections 2024-03-18T14:42:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.394+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/1169526959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e80d13a0 0x7f12e8160310 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.394+0000 7f12ee2f8640 1 -- 172.21.15.6:0/1169526959 >> 172.21.15.6:0/1169526959 conn(0x7f12e8072ba0 msgr2=0x7f12e8072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.394+0000 7f12ee2f8640 1 -- 172.21.15.6:0/1169526959 shutdown_connections 2024-03-18T14:42:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.394+0000 7f12ee2f8640 1 -- 172.21.15.6:0/1169526959 wait complete. 2024-03-18T14:42:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.395+0000 7f12ee2f8640 1 Processor -- start 2024-03-18T14:42:19.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.395+0000 7f12ee2f8640 1 -- start start 2024-03-18T14:42:19.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ee2f8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12e80b2cf0 0x7f12e81576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ee2f8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12e80d13a0 0x7f12e815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ee2f8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8145050 0x7f12e80b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ee2f8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f12e80d0950 con 0x7f12e8145050 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ee2f8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f12e8160850 con 0x7f12e80b2cf0 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ee2f8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f12e80d0e80 con 0x7f12e80d13a0 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8145050 0x7f12e80b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8145050 0x7f12e80b1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55650/0 (socket says 172.21.15.6:55650) 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 -- 172.21.15.6:0/867919365 learned_addr learned my addr 172.21.15.6:0/867919365 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12e7fff640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12e80b2cf0 0x7f12e81576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12e77fe640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12e80d13a0 0x7f12e815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 -- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12e80d13a0 msgr2=0x7f12e815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12e80d13a0 0x7f12e815eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 -- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12e80b2cf0 msgr2=0x7f12e81576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12e80b2cf0 0x7f12e81576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 -- 172.21.15.6:0/867919365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e815f4d0 con 0x7f12e8145050 2024-03-18T14:42:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ec895640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8145050 0x7f12e80b1610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009d90 tx=0x7f12e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12e4ff9640 1 -- 172.21.15.6:0/867919365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0039050 con 0x7f12e8145050 2024-03-18T14:42:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12e4ff9640 1 -- 172.21.15.6:0/867919365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e0045070 con 0x7f12e8145050 2024-03-18T14:42:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e815f7b0 con 0x7f12e8145050 2024-03-18T14:42:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.396+0000 7f12e4ff9640 1 -- 172.21.15.6:0/867919365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e003e830 con 0x7f12e8145050 2024-03-18T14:42:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.397+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e8157be0 con 0x7f12e8145050 2024-03-18T14:42:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.397+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e80751b0 con 0x7f12e8145050 2024-03-18T14:42:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.398+0000 7f12e4ff9640 1 -- 172.21.15.6:0/867919365 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f12e003e9d0 con 0x7f12e8145050 2024-03-18T14:42:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.398+0000 7f12e4ff9640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d406bbd0 0x7f12d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.398+0000 7f12e4ff9640 1 -- 172.21.15.6:0/867919365 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f12e00adb80 con 0x7f12e8145050 2024-03-18T14:42:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.398+0000 7f12e7fff640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d406bbd0 0x7f12d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.399+0000 7f12e7fff640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d406bbd0 0x7f12d406e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12dc007f70 tx=0x7f12dc007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.402+0000 7f12e4ff9640 1 -- 172.21.15.6:0/867919365 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f12e007bca0 con 0x7f12e8145050 2024-03-18T14:42:19.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.588+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f12e806eeb0 con 0x7f12e8145050 2024-03-18T14:42:19.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.588+0000 7f12e4ff9640 1 -- 172.21.15.6:0/867919365 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f12e007fd40 con 0x7f12e8145050 2024-03-18T14:42:19.591 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d406bbd0 msgr2=0x7f12d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d406bbd0 0x7f12d406e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12dc007f70 tx=0x7f12dc007f00 comp rx=0 tx=0).stop 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8145050 msgr2=0x7f12e80b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8145050 0x7f12e80b1610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009d90 tx=0x7f12e0004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 shutdown_connections 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d406bbd0 0x7f12d406e090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8145050 0x7f12e80b1610 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12e80d13a0 0x7f12e815eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 --2- 172.21.15.6:0/867919365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12e80b2cf0 0x7f12e81576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 >> 172.21.15.6:0/867919365 conn(0x7f12e8072ba0 msgr2=0x7f12e8140140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:19.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 shutdown_connections 2024-03-18T14:42:19.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.590+0000 7f12ee2f8640 1 -- 172.21.15.6:0/867919365 wait complete. 2024-03-18T14:42:19.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:19.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.706+0000 7f8b0b8d3640 1 Processor -- start 2024-03-18T14:42:19.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.706+0000 7f8b0b8d3640 1 -- start start 2024-03-18T14:42:19.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b0b8d3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040ce9d0 0x7f8b040cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b0b8d3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b040d2510 0x7f8b040d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b0b8d3640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b040d2970 con 0x7f8b040d0560 2024-03-18T14:42:19.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b0b8d3640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8b040d09d0 con 0x7f8b040d2510 2024-03-18T14:42:19.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b0b8d3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8b040cd8d0 con 0x7f8b040ce9d0 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b08e6e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b040d2510 0x7f8b040d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b08e6e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b040d2510 0x7f8b040d0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55658/0 (socket says 172.21.15.6:55658) 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b08e6e640 1 -- 172.21.15.6:0/1170413405 learned_addr learned my addr 172.21.15.6:0/1170413405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.707+0000 7f8b0966f640 1 --2- 172.21.15.6:0/1170413405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040ce9d0 0x7f8b040cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.708+0000 7f8b08e6e640 1 -- 172.21.15.6:0/1170413405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040ce9d0 msgr2=0x7f8b040cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.708+0000 7f8b08e6e640 1 --2- 172.21.15.6:0/1170413405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040ce9d0 0x7f8b040cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.708+0000 7f8b08e6e640 1 -- 172.21.15.6:0/1170413405 >> v1:172.21.15.92:6789/0 conn(0x7f8b040d0560 legacy=0x7f8b0415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.708+0000 7f8b08e6e640 1 -- 172.21.15.6:0/1170413405 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b040cfac0 con 0x7f8b040d2510 2024-03-18T14:42:19.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.708+0000 7f8b08e6e640 1 --2- 172.21.15.6:0/1170413405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040d2510 0x7f8b040d0020 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009870 tx=0x7f8af802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2c29f28a3c8a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.708+0000 7f8b03fff640 1 -- 172.21.15.6:0/1170413405 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8035020 con 0x7f8b040d2510 2024-03-18T14:42:19.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b03fff640 1 -- 172.21.15.6:0/1170413405 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af8031640 con 0x7f8b040d2510 2024-03-18T14:42:19.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b03fff640 1 -- 172.21.15.6:0/1170413405 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af802d040 con 0x7f8b040d2510 2024-03-18T14:42:19.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/1170413405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040d2510 msgr2=0x7f8b040d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/1170413405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040d2510 0x7f8b040d0020 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009870 tx=0x7f8af802b8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/1170413405 shutdown_connections 2024-03-18T14:42:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/1170413405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040d2510 0x7f8b040d0020 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/1170413405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040ce9d0 0x7f8b040cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/1170413405 >> 172.21.15.6:0/1170413405 conn(0x7f8b0408b0c0 msgr2=0x7f8b040d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/1170413405 shutdown_connections 2024-03-18T14:42:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.709+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/1170413405 wait complete. 2024-03-18T14:42:19.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.710+0000 7f8b0b8d3640 1 Processor -- start 2024-03-18T14:42:19.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.710+0000 7f8b0b8d3640 1 -- start start 2024-03-18T14:42:19.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0b8d3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040ce9d0 0x7f8b04156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0b8d3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040d0560 0x7f8b040b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0b8d3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b040d2510 0x7f8b040b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0b8d3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8b040cd280 con 0x7f8b040ce9d0 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0b8d3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b040ccfa0 con 0x7f8b040d2510 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0b8d3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8b040cd560 con 0x7f8b040d0560 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0966f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040ce9d0 0x7f8b04156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0966f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040ce9d0 0x7f8b04156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55674/0 (socket says 172.21.15.6:55674) 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0966f640 1 -- 172.21.15.6:0/3119819114 learned_addr learned my addr 172.21.15.6:0/3119819114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b08e6e640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040d0560 0x7f8b040b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b09e70640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b040d2510 0x7f8b040b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0966f640 1 -- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040d0560 msgr2=0x7f8b040b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.711+0000 7f8b0966f640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040d0560 0x7f8b040b7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b0966f640 1 -- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b040d2510 msgr2=0x7f8b040b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b0966f640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b040d2510 0x7f8b040b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b0966f640 1 -- 172.21.15.6:0/3119819114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b040b77a0 con 0x7f8b040ce9d0 2024-03-18T14:42:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b09e70640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b040d2510 0x7f8b040b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:19.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b0966f640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040ce9d0 0x7f8b04156d00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8af400efc0 tx=0x7f8af400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b01ffb640 1 -- 172.21.15.6:0/3119819114 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4018020 con 0x7f8b040ce9d0 2024-03-18T14:42:19.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b01ffb640 1 -- 172.21.15.6:0/3119819114 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af4014480 con 0x7f8b040ce9d0 2024-03-18T14:42:19.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b04157610 con 0x7f8b040ce9d0 2024-03-18T14:42:19.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b01ffb640 1 -- 172.21.15.6:0/3119819114 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4014850 con 0x7f8b040ce9d0 2024-03-18T14:42:19.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.712+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b040b6f60 con 0x7f8b040ce9d0 2024-03-18T14:42:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.713+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b040751b0 con 0x7f8b040ce9d0 2024-03-18T14:42:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.714+0000 7f8b01ffb640 1 -- 172.21.15.6:0/3119819114 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8af401b020 con 0x7f8b040ce9d0 2024-03-18T14:42:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.714+0000 7f8b01ffb640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8af006bbd0 0x7f8af006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.714+0000 7f8b01ffb640 1 -- 172.21.15.6:0/3119819114 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8af408bc40 con 0x7f8b040ce9d0 2024-03-18T14:42:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.714+0000 7f8b08e6e640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8af006bbd0 0x7f8af006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.715+0000 7f8b08e6e640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8af006bbd0 0x7f8af006e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8af8036000 tx=0x7f8af803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:19.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.718+0000 7f8b01ffb640 1 -- 172.21.15.6:0/3119819114 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8af4022020 con 0x7f8b040ce9d0 2024-03-18T14:42:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.898+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8b040cfe00 con 0x7f8b040ce9d0 2024-03-18T14:42:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.899+0000 7f8b01ffb640 1 -- 172.21.15.6:0/3119819114 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f8af4010040 con 0x7f8b040ce9d0 2024-03-18T14:42:19.902 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8af006bbd0 msgr2=0x7f8af006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8af006bbd0 0x7f8af006e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8af8036000 tx=0x7f8af803c040 comp rx=0 tx=0).stop 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040ce9d0 msgr2=0x7f8b04156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040ce9d0 0x7f8b04156d00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8af400efc0 tx=0x7f8af400c490 comp rx=0 tx=0).stop 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 shutdown_connections 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8af006bbd0 0x7f8af006e090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b040d2510 0x7f8b040b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b040d0560 0x7f8b040b7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 --2- 172.21.15.6:0/3119819114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b040ce9d0 0x7f8b04156d00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 >> 172.21.15.6:0/3119819114 conn(0x7f8b0408b0c0 msgr2=0x7f8b040af7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 shutdown_connections 2024-03-18T14:42:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:19.901+0000 7f8b0b8d3640 1 -- 172.21.15.6:0/3119819114 wait complete. 2024-03-18T14:42:19.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:20.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.016+0000 7f49e6d1b640 1 Processor -- start 2024-03-18T14:42:20.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.016+0000 7f49e6d1b640 1 -- start start 2024-03-18T14:42:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49e6d1b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f49e00b1980 0x7f49e015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49e6d1b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f49e00d1400 0x7f49e01658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49e6d1b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49e0156760 con 0x7f49e00cc670 2024-03-18T14:42:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49e6d1b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f49e00b0380 con 0x7f49e00b1980 2024-03-18T14:42:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49e6d1b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f49e015c8d0 con 0x7f49e00d1400 2024-03-18T14:42:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49e4ab7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f49e00b1980 0x7f49e015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49dffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f49e00cc670 0x7f49e00b1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55026/0 (socket says 172.21.15.6:55026) 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49dffff640 1 -- 172.21.15.6:0/3540015755 learned_addr learned my addr 172.21.15.6:0/3540015755 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.017+0000 7f49e52b8640 1 --2- 172.21.15.6:0/3540015755 >> v2:172.21.15.92:3300/0 conn(0x7f49e00d1400 0x7f49e01658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.018+0000 7f49e4ab7640 1 -- 172.21.15.6:0/3540015755 >> v2:172.21.15.92:3300/0 conn(0x7f49e00d1400 msgr2=0x7f49e01658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.018+0000 7f49e4ab7640 1 --2- 172.21.15.6:0/3540015755 >> v2:172.21.15.92:3300/0 conn(0x7f49e00d1400 0x7f49e01658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.018+0000 7f49e4ab7640 1 -- 172.21.15.6:0/3540015755 >> v1:172.21.15.6:6789/0 conn(0x7f49e00cc670 legacy=0x7f49e00b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.018+0000 7f49e4ab7640 1 -- 172.21.15.6:0/3540015755 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e015cbb0 con 0x7f49e00b1980 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.018+0000 7f49df7fe640 1 -- 172.21.15.6:0/3540015755 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599191627 0 0) 0x7f49d4001110 con 0x7f49e00cc670 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.018+0000 7f49e52b8640 1 --2- 172.21.15.6:0/3540015755 >> v2:172.21.15.92:3300/0 conn(0x7f49e00d1400 0x7f49e01658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.018+0000 7f49e4ab7640 1 --2- 172.21.15.6:0/3540015755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e015c280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f49e0072200 tx=0x7f49d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228ebd510823d329 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49df7fe640 1 -- 172.21.15.6:0/3540015755 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8039070 con 0x7f49e00b1980 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49df7fe640 1 -- 172.21.15.6:0/3540015755 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d802cc80 con 0x7f49e00b1980 2024-03-18T14:42:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49df7fe640 1 -- 172.21.15.6:0/3540015755 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8034660 con 0x7f49e00b1980 2024-03-18T14:42:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3540015755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 msgr2=0x7f49e015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3540015755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e015c280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f49e0072200 tx=0x7f49d802bed0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3540015755 shutdown_connections 2024-03-18T14:42:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3540015755 >> v2:172.21.15.92:3300/0 conn(0x7f49e00d1400 0x7f49e01658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3540015755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e015c280 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3540015755 >> 172.21.15.6:0/3540015755 conn(0x7f49e0099a50 msgr2=0x7f49e00d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3540015755 shutdown_connections 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.019+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3540015755 wait complete. 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.020+0000 7f49e6d1b640 1 Processor -- start 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.020+0000 7f49e6d1b640 1 -- start start 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e6d1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e0158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e6d1b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49e00cc670 0x7f49e00b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e6d1b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49e00d1400 0x7f49e014c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e6d1b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f49e015d170 con 0x7f49e00b1980 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e6d1b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49e009a170 con 0x7f49e00cc670 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e6d1b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f49e006bb40 con 0x7f49e00d1400 2024-03-18T14:42:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e0158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e0158fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55688/0 (socket says 172.21.15.6:55688) 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 -- 172.21.15.6:0/3861581424 learned_addr learned my addr 172.21.15.6:0/3861581424 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e52b8640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49e00d1400 0x7f49e014c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49dffff640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49e00cc670 0x7f49e00b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 -- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49e00d1400 msgr2=0x7f49e014c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49e00d1400 0x7f49e014c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 -- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49e00cc670 msgr2=0x7f49e00b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49e00cc670 0x7f49e00b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.021+0000 7f49e4ab7640 1 -- 172.21.15.6:0/3861581424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e0159680 con 0x7f49e00b1980 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.022+0000 7f49dffff640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49e00cc670 0x7f49e00b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.022+0000 7f49e4ab7640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e0158fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e80 tx=0x7f49d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.022+0000 7f49dd7fa640 1 -- 172.21.15.6:0/3861581424 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8039050 con 0x7f49e00b1980 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.022+0000 7f49dd7fa640 1 -- 172.21.15.6:0/3861581424 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d803a040 con 0x7f49e00b1980 2024-03-18T14:42:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.022+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e0157350 con 0x7f49e00b1980 2024-03-18T14:42:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.022+0000 7f49dd7fa640 1 -- 172.21.15.6:0/3861581424 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8009450 con 0x7f49e00b1980 2024-03-18T14:42:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.022+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e0157910 con 0x7f49e00b1980 2024-03-18T14:42:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.023+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49e00751b0 con 0x7f49e00b1980 2024-03-18T14:42:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.023+0000 7f49dd7fa640 1 -- 172.21.15.6:0/3861581424 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f49d8047020 con 0x7f49e00b1980 2024-03-18T14:42:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.024+0000 7f49dd7fa640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49cc06bbd0 0x7f49cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.024+0000 7f49dd7fa640 1 -- 172.21.15.6:0/3861581424 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f49d80ad030 con 0x7f49e00b1980 2024-03-18T14:42:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.024+0000 7f49dffff640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49cc06bbd0 0x7f49cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.024+0000 7f49dffff640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49cc06bbd0 0x7f49cc06e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f49e015c6e0 tx=0x7f49d4007a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.028+0000 7f49dd7fa640 1 -- 172.21.15.6:0/3861581424 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f49d804e020 con 0x7f49e00b1980 2024-03-18T14:42:20.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.208+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f49e00ce1f0 con 0x7f49e00b1980 2024-03-18T14:42:20.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.208+0000 7f49dd7fa640 1 -- 172.21.15.6:0/3861581424 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f49d807b7b0 con 0x7f49e00b1980 2024-03-18T14:42:20.211 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49cc06bbd0 msgr2=0x7f49cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49cc06bbd0 0x7f49cc06e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f49e015c6e0 tx=0x7f49d4007a10 comp rx=0 tx=0).stop 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 msgr2=0x7f49e0158fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e0158fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e80 tx=0x7f49d8004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 shutdown_connections 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f49cc06bbd0 0x7f49cc06e090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49e00d1400 0x7f49e014c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49e00cc670 0x7f49e00b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 --2- 172.21.15.6:0/3861581424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49e00b1980 0x7f49e0158fa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 >> 172.21.15.6:0/3861581424 conn(0x7f49e0099a50 msgr2=0x7f49e015d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 shutdown_connections 2024-03-18T14:42:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.210+0000 7f49e6d1b640 1 -- 172.21.15.6:0/3861581424 wait complete. 2024-03-18T14:42:20.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:20.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.323+0000 7fcf6d7c4640 1 Processor -- start 2024-03-18T14:42:20.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.323+0000 7fcf6d7c4640 1 -- start start 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.323+0000 7fcf6d7c4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcf680d0d90 0x7fcf680b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.323+0000 7fcf6d7c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf680b9ae0 con 0x7fcf680b9650 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.323+0000 7fcf6d7c4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf680cef50 con 0x7fcf6815e170 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.323+0000 7fcf6d7c4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcf680cf230 con 0x7fcf680d0d90 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf667fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcf680b9650 0x7fcf680d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55032/0 (socket says 172.21.15.6:55032) 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf667fc640 1 -- 172.21.15.6:0/1163684588 learned_addr learned my addr 172.21.15.6:0/1163684588 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf677fe640 1 --2- 172.21.15.6:0/1163684588 >> v2:172.21.15.92:3300/0 conn(0x7fcf680d0d90 0x7fcf680b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf65ffb640 1 -- 172.21.15.6:0/1163684588 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 21872893 0 0) 0x7fcf680b9ae0 con 0x7fcf680b9650 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf65ffb640 1 -- 172.21.15.6:0/1163684588 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf50003520 con 0x7fcf680b9650 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf65ffb640 1 -- 172.21.15.6:0/1163684588 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 514397585 0 0) 0x7fcf680cef50 con 0x7fcf6815e170 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf65ffb640 1 -- 172.21.15.6:0/1163684588 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf680b9ae0 con 0x7fcf6815e170 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf677fe640 1 -- 172.21.15.6:0/1163684588 >> v1:172.21.15.92:6789/0 conn(0x7fcf6815e170 legacy=0x7fcf680d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf677fe640 1 -- 172.21.15.6:0/1163684588 >> v1:172.21.15.6:6789/0 conn(0x7fcf680b9650 legacy=0x7fcf680d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.324+0000 7fcf677fe640 1 -- 172.21.15.6:0/1163684588 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf680d2b30 con 0x7fcf680d0d90 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf677fe640 1 --2- 172.21.15.6:0/1163684588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680d0d90 0x7fcf680b6dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009d90 tx=0x7fcf6002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c0a4c44a018acf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf65ffb640 1 -- 172.21.15.6:0/1163684588 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60039070 con 0x7fcf680d0d90 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf65ffb640 1 -- 172.21.15.6:0/1163684588 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf6002cd00 con 0x7fcf680d0d90 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf65ffb640 1 -- 172.21.15.6:0/1163684588 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60034640 con 0x7fcf680d0d90 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/1163684588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680d0d90 msgr2=0x7fcf680b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/1163684588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680d0d90 0x7fcf680b6dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009d90 tx=0x7fcf6002bf50 comp rx=0 tx=0).stop 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/1163684588 shutdown_connections 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.325+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/1163684588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680d0d90 0x7fcf680b6dd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.326+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/1163684588 >> 172.21.15.6:0/1163684588 conn(0x7fcf6806c900 msgr2=0x7fcf6806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.326+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/1163684588 shutdown_connections 2024-03-18T14:42:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.326+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/1163684588 wait complete. 2024-03-18T14:42:20.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.327+0000 7fcf6d7c4640 1 Processor -- start 2024-03-18T14:42:20.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.327+0000 7fcf6d7c4640 1 -- start start 2024-03-18T14:42:20.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680b9650 0x7fcf680c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf680d0d90 0x7fcf681e6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf6815e170 0x7fcf681e6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcf680d2820 con 0x7fcf680d0d90 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcf680d3a40 con 0x7fcf680b9650 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcf680ceb70 con 0x7fcf6815e170 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf680d0d90 0x7fcf681e6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf680d0d90 0x7fcf681e6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55692/0 (socket says 172.21.15.6:55692) 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 -- 172.21.15.6:0/582199765 learned_addr learned my addr 172.21.15.6:0/582199765 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf66ffd640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680b9650 0x7fcf680c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf677fe640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf6815e170 0x7fcf681e6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 -- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf6815e170 msgr2=0x7fcf681e6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf6815e170 0x7fcf681e6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 -- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680b9650 msgr2=0x7fcf680c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680b9650 0x7fcf680c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 -- 172.21.15.6:0/582199765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf680cd700 con 0x7fcf680d0d90 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf667fc640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf680d0d90 0x7fcf681e6740 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c530 tx=0x7fcf5c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf4f7fe640 1 -- 172.21.15.6:0/582199765 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c017070 con 0x7fcf680d0d90 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf4f7fe640 1 -- 172.21.15.6:0/582199765 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf5c005d00 con 0x7fcf680d0d90 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf680cd9e0 con 0x7fcf680d0d90 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf4f7fe640 1 -- 172.21.15.6:0/582199765 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0072b0 con 0x7fcf680d0d90 2024-03-18T14:42:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.328+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6814add0 con 0x7fcf680d0d90 2024-03-18T14:42:20.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.329+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf680ce840 con 0x7fcf680d0d90 2024-03-18T14:42:20.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.330+0000 7fcf4f7fe640 1 -- 172.21.15.6:0/582199765 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcf5c01a070 con 0x7fcf680d0d90 2024-03-18T14:42:20.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.330+0000 7fcf4f7fe640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcf5406bbd0 0x7fcf5406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.330+0000 7fcf4f7fe640 1 -- 172.21.15.6:0/582199765 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcf5c08a180 con 0x7fcf680d0d90 2024-03-18T14:42:20.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.331+0000 7fcf66ffd640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcf5406bbd0 0x7fcf5406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.331+0000 7fcf66ffd640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcf5406bbd0 0x7fcf5406e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fd0 tx=0x7fcf58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.335+0000 7fcf4f7fe640 1 -- 172.21.15.6:0/582199765 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcf680ce840 con 0x7fcf680d0d90 2024-03-18T14:42:20.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.517+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf680b8fb0 con 0x7fcf680d0d90 2024-03-18T14:42:20.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.518+0000 7fcf4f7fe640 1 -- 172.21.15.6:0/582199765 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fcf5c058370 con 0x7fcf680d0d90 2024-03-18T14:42:20.521 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:20.521 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcf5406bbd0 msgr2=0x7fcf5406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcf5406bbd0 0x7fcf5406e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fd0 tx=0x7fcf58009040 comp rx=0 tx=0).stop 2024-03-18T14:42:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf680d0d90 msgr2=0x7fcf681e6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf680d0d90 0x7fcf681e6740 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c530 tx=0x7fcf5c00c9f0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 shutdown_connections 2024-03-18T14:42:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcf5406bbd0 0x7fcf5406e090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf6815e170 0x7fcf681e6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf680d0d90 0x7fcf681e6740 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 --2- 172.21.15.6:0/582199765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf680b9650 0x7fcf680c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 >> 172.21.15.6:0/582199765 conn(0x7fcf6806c900 msgr2=0x7fcf680b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:20.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 shutdown_connections 2024-03-18T14:42:20.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.520+0000 7fcf6d7c4640 1 -- 172.21.15.6:0/582199765 wait complete. 2024-03-18T14:42:20.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:20.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:20.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.636+0000 7faa4c3f4640 1 Processor -- start 2024-03-18T14:42:20.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.636+0000 7faa4c3f4640 1 -- start start 2024-03-18T14:42:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.637+0000 7faa4c3f4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4406b810 0x7faa440ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.637+0000 7faa4c3f4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa440b22c0 con 0x7faa4415d640 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.637+0000 7faa4c3f4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4415e650 con 0x7faa4406b430 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.637+0000 7faa4c3f4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faa4415e930 con 0x7faa4406b810 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.637+0000 7faa4a991640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa4415d640 0x7faa4415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55042/0 (socket says 172.21.15.6:55042) 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.637+0000 7faa4a991640 1 -- 172.21.15.6:0/2402898418 learned_addr learned my addr 172.21.15.6:0/2402898418 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4998f640 1 --2- 172.21.15.6:0/2402898418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4406b810 0x7faa440ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4918e640 1 -- 172.21.15.6:0/2402898418 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124525601 0 0) 0x7faa440b22c0 con 0x7faa4415d640 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4918e640 1 -- 172.21.15.6:0/2402898418 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa30003520 con 0x7faa4415d640 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4918e640 1 -- 172.21.15.6:0/2402898418 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114092571 0 0) 0x7faa4415e650 con 0x7faa4406b430 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4918e640 1 -- 172.21.15.6:0/2402898418 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa440b22c0 con 0x7faa4406b430 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4998f640 1 -- 172.21.15.6:0/2402898418 >> v1:172.21.15.92:6789/0 conn(0x7faa4406b430 legacy=0x7faa440b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4998f640 1 -- 172.21.15.6:0/2402898418 >> v1:172.21.15.6:6789/0 conn(0x7faa4415d640 legacy=0x7faa4415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.638+0000 7faa4998f640 1 -- 172.21.15.6:0/2402898418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4415ec10 con 0x7faa4406b810 2024-03-18T14:42:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4998f640 1 --2- 172.21.15.6:0/2402898418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4406b810 0x7faa440ce670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faa340097d0 tx=0x7faa3402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8efcbf20ab18b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4918e640 1 -- 172.21.15.6:0/2402898418 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34035020 con 0x7faa4406b810 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4918e640 1 -- 172.21.15.6:0/2402898418 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa34031890 con 0x7faa4406b810 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4918e640 1 -- 172.21.15.6:0/2402898418 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34031c60 con 0x7faa4406b810 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4c3f4640 1 -- 172.21.15.6:0/2402898418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4406b810 msgr2=0x7faa440ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/2402898418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4406b810 0x7faa440ce670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faa340097d0 tx=0x7faa3402b930 comp rx=0 tx=0).stop 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4c3f4640 1 -- 172.21.15.6:0/2402898418 shutdown_connections 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/2402898418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4406b810 0x7faa440ce670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4c3f4640 1 -- 172.21.15.6:0/2402898418 >> 172.21.15.6:0/2402898418 conn(0x7faa44099a50 msgr2=0x7faa440d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4c3f4640 1 -- 172.21.15.6:0/2402898418 shutdown_connections 2024-03-18T14:42:20.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.639+0000 7faa4c3f4640 1 -- 172.21.15.6:0/2402898418 wait complete. 2024-03-18T14:42:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.640+0000 7faa4c3f4640 1 Processor -- start 2024-03-18T14:42:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4c3f4640 1 -- start start 2024-03-18T14:42:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4c3f4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa4406b430 0x7faa44155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4c3f4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4406b810 0x7faa440c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4c3f4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4415d640 0x7faa440b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4c3f4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faa440cce70 con 0x7faa4406b810 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4c3f4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faa440cccf0 con 0x7faa4406b430 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4c3f4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faa4415e1d0 con 0x7faa4415d640 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4998f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4406b810 0x7faa440c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4998f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4406b810 0x7faa440c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55696/0 (socket says 172.21.15.6:55696) 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4998f640 1 -- 172.21.15.6:0/3829213815 learned_addr learned my addr 172.21.15.6:0/3829213815 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4a991640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4415d640 0x7faa440b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.641+0000 7faa4a190640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa4406b430 0x7faa44155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4998f640 1 -- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4415d640 msgr2=0x7faa440b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4998f640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4415d640 0x7faa440b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4998f640 1 -- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa4406b430 msgr2=0x7faa44155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4998f640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa4406b430 0x7faa44155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4998f640 1 -- 172.21.15.6:0/3829213815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa440c83e0 con 0x7faa4406b810 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4998f640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4406b810 0x7faa440c7a90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faa340098f0 tx=0x7faa340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa3affd640 1 -- 172.21.15.6:0/3829213815 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34035040 con 0x7faa4406b810 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa3affd640 1 -- 172.21.15.6:0/3829213815 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa34030d20 con 0x7faa4406b810 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa440c86c0 con 0x7faa4406b810 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa3affd640 1 -- 172.21.15.6:0/3829213815 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34006e10 con 0x7faa4406b810 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.642+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4415aa80 con 0x7faa4406b810 2024-03-18T14:42:20.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.643+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa440751b0 con 0x7faa4406b810 2024-03-18T14:42:20.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.643+0000 7faa3affd640 1 -- 172.21.15.6:0/3829213815 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faa34038020 con 0x7faa4406b810 2024-03-18T14:42:20.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.644+0000 7faa3affd640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2806bbd0 0x7faa2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.644+0000 7faa3affd640 1 -- 172.21.15.6:0/3829213815 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7faa340ad320 con 0x7faa4406b810 2024-03-18T14:42:20.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.644+0000 7faa4a190640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2806bbd0 0x7faa2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.644+0000 7faa4a190640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2806bbd0 0x7faa2806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faa40007fd0 tx=0x7faa40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.648+0000 7faa3affd640 1 -- 172.21.15.6:0/3829213815 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faa3402d040 con 0x7faa4406b810 2024-03-18T14:42:20.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.830+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa440d1380 con 0x7faa4406b810 2024-03-18T14:42:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.830+0000 7faa3affd640 1 -- 172.21.15.6:0/3829213815 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7faa3407b450 con 0x7faa4406b810 2024-03-18T14:42:20.833 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2806bbd0 msgr2=0x7faa2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2806bbd0 0x7faa2806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faa40007fd0 tx=0x7faa40009040 comp rx=0 tx=0).stop 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4406b810 msgr2=0x7faa440c7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4406b810 0x7faa440c7a90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faa340098f0 tx=0x7faa340046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 shutdown_connections 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2806bbd0 0x7faa2806e090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa4415d640 0x7faa440b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4406b810 0x7faa440c7a90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 --2- 172.21.15.6:0/3829213815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa4406b430 0x7faa44155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 >> 172.21.15.6:0/3829213815 conn(0x7faa44099a50 msgr2=0x7faa440cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 shutdown_connections 2024-03-18T14:42:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.832+0000 7faa4c3f4640 1 -- 172.21.15.6:0/3829213815 wait complete. 2024-03-18T14:42:20.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:20.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.947+0000 7f9a73a0b640 1 Processor -- start 2024-03-18T14:42:20.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.948+0000 7f9a73a0b640 1 -- start start 2024-03-18T14:42:20.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.948+0000 7f9a73a0b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9a6c161740 0x7f9a6c09ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.948+0000 7f9a73a0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c06cef0 0x7f9a6c099cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.948+0000 7f9a73a0b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c0b4920 con 0x7f9a6c09a1f0 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.948+0000 7f9a73a0b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9a6c0cc120 con 0x7f9a6c161740 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.948+0000 7f9a73a0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9a6c06d740 con 0x7f9a6c06cef0 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a71fa8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a6c09a1f0 0x7f9a6c0cfe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55048/0 (socket says 172.21.15.6:55048) 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a71fa8640 1 -- 172.21.15.6:0/537340778 learned_addr learned my addr 172.21.15.6:0/537340778 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a717a7640 1 --2- 172.21.15.6:0/537340778 >> v2:172.21.15.92:3300/0 conn(0x7f9a6c161740 0x7f9a6c09ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a70fa6640 1 --2- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c06cef0 0x7f9a6c099cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a63fff640 1 -- 172.21.15.6:0/537340778 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294311172 0 0) 0x7f9a64001490 con 0x7f9a6c09a1f0 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a63fff640 1 -- 172.21.15.6:0/537340778 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a58003520 con 0x7f9a6c09a1f0 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a717a7640 1 -- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c06cef0 msgr2=0x7f9a6c099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a717a7640 1 --2- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c06cef0 0x7f9a6c099cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a717a7640 1 -- 172.21.15.6:0/537340778 >> v1:172.21.15.6:6789/0 conn(0x7f9a6c09a1f0 legacy=0x7f9a6c0cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a717a7640 1 -- 172.21.15.6:0/537340778 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c0cc370 con 0x7f9a6c161740 2024-03-18T14:42:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.949+0000 7f9a70fa6640 1 --2- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c06cef0 0x7f9a6c099cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a717a7640 1 --2- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 0x7f9a6c09ad30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9a680097a0 tx=0x7f9a6802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ddd89d17f927888 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a63fff640 1 -- 172.21.15.6:0/537340778 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68035020 con 0x7f9a6c161740 2024-03-18T14:42:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a63fff640 1 -- 172.21.15.6:0/537340778 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a68031830 con 0x7f9a6c161740 2024-03-18T14:42:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a63fff640 1 -- 172.21.15.6:0/537340778 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68031c00 con 0x7f9a6c161740 2024-03-18T14:42:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a73a0b640 1 -- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 msgr2=0x7f9a6c09ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 0x7f9a6c09ad30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9a680097a0 tx=0x7f9a6802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a73a0b640 1 -- 172.21.15.6:0/537340778 shutdown_connections 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c06cef0 0x7f9a6c099cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/537340778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 0x7f9a6c09ad30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.950+0000 7f9a73a0b640 1 -- 172.21.15.6:0/537340778 >> 172.21.15.6:0/537340778 conn(0x7f9a6c0b35c0 msgr2=0x7f9a6c0b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.951+0000 7f9a73a0b640 1 -- 172.21.15.6:0/537340778 shutdown_connections 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.951+0000 7f9a73a0b640 1 -- 172.21.15.6:0/537340778 wait complete. 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 Processor -- start 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 -- start start 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a6c06cef0 0x7f9a6c0cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c09a1f0 0x7f9a6c153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 0x7f9a6c0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c0cde60 con 0x7f9a6c06cef0 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c0cc930 con 0x7f9a6c161740 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a73a0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9a6c06d330 con 0x7f9a6c09a1f0 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.952+0000 7f9a717a7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a6c06cef0 0x7f9a6c0cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a6c06cef0 0x7f9a6c0cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55708/0 (socket says 172.21.15.6:55708) 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 -- 172.21.15.6:0/382271812 learned_addr learned my addr 172.21.15.6:0/382271812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a70fa6640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c09a1f0 0x7f9a6c153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a71fa8640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 0x7f9a6c0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 -- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c09a1f0 msgr2=0x7f9a6c153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c09a1f0 0x7f9a6c153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 -- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 msgr2=0x7f9a6c0b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 0x7f9a6c0b4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 -- 172.21.15.6:0/382271812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c0c76d0 con 0x7f9a6c06cef0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a717a7640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a6c06cef0 0x7f9a6c0cb630 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9a6802bee0 tx=0x7f9a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a61ffb640 1 -- 172.21.15.6:0/382271812 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68048070 con 0x7f9a6c06cef0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a61ffb640 1 -- 172.21.15.6:0/382271812 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a6803a6f0 con 0x7f9a6c06cef0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c0c79b0 con 0x7f9a6c06cef0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a61ffb640 1 -- 172.21.15.6:0/382271812 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6803aae0 con 0x7f9a6c06cef0 2024-03-18T14:42:20.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.953+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6c0b5040 con 0x7f9a6c06cef0 2024-03-18T14:42:20.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.954+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6c0751b0 con 0x7f9a6c06cef0 2024-03-18T14:42:20.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.955+0000 7f9a61ffb640 1 -- 172.21.15.6:0/382271812 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9a6804b070 con 0x7f9a6c06cef0 2024-03-18T14:42:20.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.955+0000 7f9a61ffb640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a5006bbd0 0x7f9a5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:20.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.955+0000 7f9a61ffb640 1 -- 172.21.15.6:0/382271812 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9a680ac4a0 con 0x7f9a6c06cef0 2024-03-18T14:42:20.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.955+0000 7f9a70fa6640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a5006bbd0 0x7f9a5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:20.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.956+0000 7f9a70fa6640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a5006bbd0 0x7f9a5006e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009560 tx=0x7f9a5c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:20.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:20.960+0000 7f9a61ffb640 1 -- 172.21.15.6:0/382271812 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9a6807a5d0 con 0x7f9a6c06cef0 2024-03-18T14:42:21.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.142+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a6c0d12e0 con 0x7f9a6c06cef0 2024-03-18T14:42:21.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.142+0000 7f9a61ffb640 1 -- 172.21.15.6:0/382271812 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f9a6807e670 con 0x7f9a6c06cef0 2024-03-18T14:42:21.145 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:21.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.144+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a5006bbd0 msgr2=0x7f9a5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.144+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a5006bbd0 0x7f9a5006e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009560 tx=0x7f9a5c009190 comp rx=0 tx=0).stop 2024-03-18T14:42:21.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.144+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a6c06cef0 msgr2=0x7f9a6c0cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.144+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a6c06cef0 0x7f9a6c0cb630 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9a6802bee0 tx=0x7f9a680046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 shutdown_connections 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9a5006bbd0 0x7f9a5006e090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6c161740 0x7f9a6c0b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a6c09a1f0 0x7f9a6c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 --2- 172.21.15.6:0/382271812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a6c06cef0 0x7f9a6c0cb630 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 >> 172.21.15.6:0/382271812 conn(0x7f9a6c0b35c0 msgr2=0x7f9a6c0714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 shutdown_connections 2024-03-18T14:42:21.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.145+0000 7f9a73a0b640 1 -- 172.21.15.6:0/382271812 wait complete. 2024-03-18T14:42:21.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:21.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.260+0000 7f7bb2b37640 1 Processor -- start 2024-03-18T14:42:21.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.260+0000 7f7bb2b37640 1 -- start start 2024-03-18T14:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb2b37640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7bac0afac0 0x7f7bac06b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb2b37640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7bac06b700 0x7f7bac0cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb2b37640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bac0ce040 con 0x7f7bac15e030 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb2b37640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7bac15e410 con 0x7f7bac0afac0 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb2b37640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7bac0d11a0 con 0x7f7bac06b700 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7babfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7bac0afac0 0x7f7bac06b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb08d3640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7bac15e030 0x7f7bac0cd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55060/0 (socket says 172.21.15.6:55060) 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7babfff640 1 --2- 172.21.15.6:0/1713868887 >> v2:172.21.15.6:3300/0 conn(0x7f7bac0afac0 0x7f7bac06b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55720/0 (socket says 172.21.15.6:55720) 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb08d3640 1 -- 172.21.15.6:0/1713868887 learned_addr learned my addr 172.21.15.6:0/1713868887 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.261+0000 7f7bb10d4640 1 --2- 172.21.15.6:0/1713868887 >> v2:172.21.15.92:3300/0 conn(0x7f7bac06b700 0x7f7bac0cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.262+0000 7f7babfff640 1 -- 172.21.15.6:0/1713868887 >> v2:172.21.15.92:3300/0 conn(0x7f7bac06b700 msgr2=0x7f7bac0cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.262+0000 7f7babfff640 1 --2- 172.21.15.6:0/1713868887 >> v2:172.21.15.92:3300/0 conn(0x7f7bac06b700 0x7f7bac0cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.262+0000 7f7babfff640 1 -- 172.21.15.6:0/1713868887 >> v1:172.21.15.6:6789/0 conn(0x7f7bac15e030 legacy=0x7f7bac0cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.262+0000 7f7babfff640 1 -- 172.21.15.6:0/1713868887 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac0d0870 con 0x7f7bac0afac0 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.262+0000 7f7babfff640 1 --2- 172.21.15.6:0/1713868887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac06b1c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7bac070e40 tx=0x7f7b9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e0e6101cbf6a04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bab7fe640 1 -- 172.21.15.6:0/1713868887 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b980315a0 con 0x7f7bac0afac0 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bab7fe640 1 -- 172.21.15.6:0/1713868887 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b98037070 con 0x7f7bac0afac0 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bab7fe640 1 -- 172.21.15.6:0/1713868887 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98031960 con 0x7f7bac0afac0 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 -- 172.21.15.6:0/1713868887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 msgr2=0x7f7bac06b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/1713868887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac06b1c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7bac070e40 tx=0x7f7b9802b840 comp rx=0 tx=0).stop 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 -- 172.21.15.6:0/1713868887 shutdown_connections 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/1713868887 >> v2:172.21.15.92:3300/0 conn(0x7f7bac06b700 0x7f7bac0cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/1713868887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac06b1c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 -- 172.21.15.6:0/1713868887 >> 172.21.15.6:0/1713868887 conn(0x7f7bac099a50 msgr2=0x7f7bac099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 -- 172.21.15.6:0/1713868887 shutdown_connections 2024-03-18T14:42:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.263+0000 7f7bb2b37640 1 -- 172.21.15.6:0/1713868887 wait complete. 2024-03-18T14:42:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.264+0000 7f7bb2b37640 1 Processor -- start 2024-03-18T14:42:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.264+0000 7f7bb2b37640 1 -- start start 2024-03-18T14:42:21.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb2b37640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bac06b700 0x7f7bac0c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb2b37640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb2b37640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bac15e030 0x7f7bac15c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb2b37640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac0d1630 con 0x7f7bac0afac0 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb2b37640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac0d0e30 con 0x7f7bac15e030 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb2b37640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7bac0c5e30 con 0x7f7bac06b700 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7babfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7babfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac15be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55726/0 (socket says 172.21.15.6:55726) 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7babfff640 1 -- 172.21.15.6:0/2636246987 learned_addr learned my addr 172.21.15.6:0/2636246987 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb10d4640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bac15e030 0x7f7bac15c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.265+0000 7f7bb08d3640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bac06b700 0x7f7bac0c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7babfff640 1 -- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bac06b700 msgr2=0x7f7bac0c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7babfff640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bac06b700 0x7f7bac0c53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7babfff640 1 -- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bac15e030 msgr2=0x7f7bac15c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7babfff640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bac15e030 0x7f7bac15c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7babfff640 1 -- 172.21.15.6:0/2636246987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac0c6110 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7babfff640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac15be80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7b98000c00 tx=0x7f7b980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7ba97fa640 1 -- 172.21.15.6:0/2636246987 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98032630 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7ba97fa640 1 -- 172.21.15.6:0/2636246987 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b980327f0 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bac0c63f0 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7ba97fa640 1 -- 172.21.15.6:0/2636246987 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98032be0 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.266+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bac0c6ed0 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.267+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bac0751b0 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.267+0000 7f7ba97fa640 1 -- 172.21.15.6:0/2636246987 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7b9804c020 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.268+0000 7f7ba97fa640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b9006bbd0 0x7f7b9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.268+0000 7f7ba97fa640 1 -- 172.21.15.6:0/2636246987 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7b9802f020 con 0x7f7bac0afac0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.268+0000 7f7bb08d3640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b9006bbd0 0x7f7b9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.268+0000 7f7bb08d3640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b9006bbd0 0x7f7b9006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b5ab0 tx=0x7f7ba0007e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.272+0000 7f7ba97fa640 1 -- 172.21.15.6:0/2636246987 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7b98040070 con 0x7f7bac0afac0 2024-03-18T14:42:21.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.452+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7bac0ccae0 con 0x7f7bac0afac0 2024-03-18T14:42:21.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.452+0000 7f7ba97fa640 1 -- 172.21.15.6:0/2636246987 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f7b9807aa60 con 0x7f7bac0afac0 2024-03-18T14:42:21.455 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:21.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b9006bbd0 msgr2=0x7f7b9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b9006bbd0 0x7f7b9006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b5ab0 tx=0x7f7ba0007e80 comp rx=0 tx=0).stop 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 msgr2=0x7f7bac15be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac15be80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7b98000c00 tx=0x7f7b980046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 shutdown_connections 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b9006bbd0 0x7f7b9006e090 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bac15e030 0x7f7bac15c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac0afac0 0x7f7bac15be80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 --2- 172.21.15.6:0/2636246987 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bac06b700 0x7f7bac0c53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 >> 172.21.15.6:0/2636246987 conn(0x7f7bac099a50 msgr2=0x7f7bac0ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 shutdown_connections 2024-03-18T14:42:21.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.455+0000 7f7bb2b37640 1 -- 172.21.15.6:0/2636246987 wait complete. 2024-03-18T14:42:21.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:21.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.570+0000 7f019673b640 1 Processor -- start 2024-03-18T14:42:21.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.570+0000 7f019673b640 1 -- start start 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.570+0000 7f019673b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.570+0000 7f019673b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01900b7e00 0x7f01900b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.570+0000 7f019673b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f019015e3b0 con 0x7f01900b5740 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.570+0000 7f019673b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f01900b60e0 con 0x7f01900b7e00 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.570+0000 7f019673b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f01900b8470 con 0x7f019015dfd0 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f0194cd8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01900b7e00 0x7f01900b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f018f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f01900b5740 0x7f01900d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55068/0 (socket says 172.21.15.6:55068) 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f018f7fe640 1 -- 172.21.15.6:0/3815514670 learned_addr learned my addr 172.21.15.6:0/3815514670 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f018ffff640 1 --2- 172.21.15.6:0/3815514670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f0194cd8640 1 -- 172.21.15.6:0/3815514670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 msgr2=0x7f01900b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f0194cd8640 1 --2- 172.21.15.6:0/3815514670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f0194cd8640 1 -- 172.21.15.6:0/3815514670 >> v1:172.21.15.6:6789/0 conn(0x7f01900b5740 legacy=0x7f01900d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f0194cd8640 1 -- 172.21.15.6:0/3815514670 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01900b8750 con 0x7f01900b7e00 2024-03-18T14:42:21.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.571+0000 7f018effd640 1 -- 172.21.15.6:0/3815514670 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762280983 0 0) 0x7f019015e3b0 con 0x7f01900b5740 2024-03-18T14:42:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f0194cd8640 1 --2- 172.21.15.6:0/3815514670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01900b5ba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0188009d90 tx=0x7f018802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9947d2ff8fbe2520 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f018effd640 1 -- 172.21.15.6:0/3815514670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188039070 con 0x7f01900b7e00 2024-03-18T14:42:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f018effd640 1 -- 172.21.15.6:0/3815514670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f018802cc50 con 0x7f01900b7e00 2024-03-18T14:42:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f018effd640 1 -- 172.21.15.6:0/3815514670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188034440 con 0x7f01900b7e00 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 -- 172.21.15.6:0/3815514670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 msgr2=0x7f01900b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 --2- 172.21.15.6:0/3815514670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01900b5ba0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0188009d90 tx=0x7f018802bea0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 -- 172.21.15.6:0/3815514670 shutdown_connections 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 --2- 172.21.15.6:0/3815514670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01900b5ba0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 --2- 172.21.15.6:0/3815514670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 -- 172.21.15.6:0/3815514670 >> 172.21.15.6:0/3815514670 conn(0x7f0190072d10 msgr2=0x7f0190073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 -- 172.21.15.6:0/3815514670 shutdown_connections 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.572+0000 7f019673b640 1 -- 172.21.15.6:0/3815514670 wait complete. 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.573+0000 7f019673b640 1 Processor -- start 2024-03-18T14:42:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f019673b640 1 -- start start 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f019673b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01900b5740 0x7f019015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f019673b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01901f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f019673b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f019673b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f01900b9b90 con 0x7f01900b7e00 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f019673b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f01900b8d10 con 0x7f01900b5740 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f019673b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f01900d2550 con 0x7f019015dfd0 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f018f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01901f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f018f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01901f3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55744/0 (socket says 172.21.15.6:55744) 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.574+0000 7f018f7fe640 1 -- 172.21.15.6:0/2788314069 learned_addr learned my addr 172.21.15.6:0/2788314069 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f0194cd8640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018ffff640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01900b5740 0x7f019015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018f7fe640 1 -- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 msgr2=0x7f01900c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018f7fe640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018f7fe640 1 -- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01900b5740 msgr2=0x7f019015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018f7fe640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01900b5740 0x7f019015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018f7fe640 1 -- 172.21.15.6:0/2788314069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019015d0b0 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018f7fe640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01901f3a20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f018400c4d0 tx=0x7f018400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018cff9640 1 -- 172.21.15.6:0/2788314069 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184018070 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018cff9640 1 -- 172.21.15.6:0/2788314069 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184007500 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01900c82b0 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f018cff9640 1 -- 172.21.15.6:0/2788314069 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184004a20 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.575+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01900c8870 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.576+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01900751b0 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.576+0000 7f018cff9640 1 -- 172.21.15.6:0/2788314069 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f018401b070 con 0x7f01900b7e00 2024-03-18T14:42:21.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.577+0000 7f018cff9640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f017c06bbd0 0x7f017c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.577+0000 7f018cff9640 1 -- 172.21.15.6:0/2788314069 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f018408a030 con 0x7f01900b7e00 2024-03-18T14:42:21.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.577+0000 7f018ffff640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f017c06bbd0 0x7f017c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.577+0000 7f018ffff640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f017c06bbd0 0x7f017c06e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0180009aa0 tx=0x7f0180009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.581+0000 7f018cff9640 1 -- 172.21.15.6:0/2788314069 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0184058160 con 0x7f01900b7e00 2024-03-18T14:42:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.763+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f01900ce140 con 0x7f01900b7e00 2024-03-18T14:42:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.763+0000 7f018cff9640 1 -- 172.21.15.6:0/2788314069 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f018405c200 con 0x7f01900b7e00 2024-03-18T14:42:21.766 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f017c06bbd0 msgr2=0x7f017c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f017c06bbd0 0x7f017c06e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0180009aa0 tx=0x7f0180009190 comp rx=0 tx=0).stop 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 msgr2=0x7f01901f3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01901f3a20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f018400c4d0 tx=0x7f018400c990 comp rx=0 tx=0).stop 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 shutdown_connections 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f017c06bbd0 0x7f017c06e090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f019015dfd0 0x7f01900c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01900b7e00 0x7f01901f3a20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 --2- 172.21.15.6:0/2788314069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01900b5740 0x7f019015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 >> 172.21.15.6:0/2788314069 conn(0x7f0190072d10 msgr2=0x7f01900b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 shutdown_connections 2024-03-18T14:42:21.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.765+0000 7f019673b640 1 -- 172.21.15.6:0/2788314069 wait complete. 2024-03-18T14:42:21.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:21.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.878+0000 7fbf6faf1640 1 Processor -- start 2024-03-18T14:42:21.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6faf1640 1 -- start start 2024-03-18T14:42:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6faf1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf6815e020 0x7fbf68162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6faf1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf680d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6faf1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf680b9660 con 0x7fbf680ced00 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6faf1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbf680d2270 con 0x7fbf6815e020 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6faf1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbf680ce390 con 0x7fbf680d3210 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6d88d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf6815e020 0x7fbf68162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6d88d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbf6815e020 0x7fbf68162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55760/0 (socket says 172.21.15.6:55760) 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.879+0000 7fbf6d88d640 1 -- 172.21.15.6:0/239324721 learned_addr learned my addr 172.21.15.6:0/239324721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.880+0000 7fbf6e08e640 1 --2- 172.21.15.6:0/239324721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf680d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.880+0000 7fbf6d88d640 1 -- 172.21.15.6:0/239324721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 msgr2=0x7fbf680d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.880+0000 7fbf6d88d640 1 --2- 172.21.15.6:0/239324721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf680d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.880+0000 7fbf6d88d640 1 -- 172.21.15.6:0/239324721 >> v1:172.21.15.92:6789/0 conn(0x7fbf680ced00 legacy=0x7fbf680d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.880+0000 7fbf6c88b640 1 -- 172.21.15.6:0/239324721 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104430695 0 0) 0x7fbf680b9660 con 0x7fbf680ced00 2024-03-18T14:42:21.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.880+0000 7fbf6d88d640 1 -- 172.21.15.6:0/239324721 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf680b9900 con 0x7fbf6815e020 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.880+0000 7fbf6d88d640 1 --2- 172.21.15.6:0/239324721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf6815e020 0x7fbf68162380 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbf680b9830 tx=0x7fbf6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b63cce33cb2647b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6c88b640 1 -- 172.21.15.6:0/239324721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64035020 con 0x7fbf6815e020 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6c88b640 1 -- 172.21.15.6:0/239324721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf64031640 con 0x7fbf6815e020 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6c88b640 1 -- 172.21.15.6:0/239324721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6402d040 con 0x7fbf6815e020 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 -- 172.21.15.6:0/239324721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf6815e020 msgr2=0x7fbf68162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/239324721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf6815e020 0x7fbf68162380 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbf680b9830 tx=0x7fbf6402b8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 -- 172.21.15.6:0/239324721 shutdown_connections 2024-03-18T14:42:21.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/239324721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf680d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/239324721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf6815e020 0x7fbf68162380 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 -- 172.21.15.6:0/239324721 >> 172.21.15.6:0/239324721 conn(0x7fbf68072ce0 msgr2=0x7fbf680730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 -- 172.21.15.6:0/239324721 shutdown_connections 2024-03-18T14:42:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.881+0000 7fbf6faf1640 1 -- 172.21.15.6:0/239324721 wait complete. 2024-03-18T14:42:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.882+0000 7fbf6faf1640 1 Processor -- start 2024-03-18T14:42:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.882+0000 7fbf6faf1640 1 -- start start 2024-03-18T14:42:21.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6faf1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf680ced00 0x7fbf68153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6faf1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf681f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6faf1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf6815e020 0x7fbf681f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6faf1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbf680cf2c0 con 0x7fbf680ced00 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6faf1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbf680ce890 con 0x7fbf6815e020 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6faf1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbf680cf440 con 0x7fbf680d3210 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6d88d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf680ced00 0x7fbf68153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6d88d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf680ced00 0x7fbf68153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55774/0 (socket says 172.21.15.6:55774) 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6d88d640 1 -- 172.21.15.6:0/2413633724 learned_addr learned my addr 172.21.15.6:0/2413633724 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6e08e640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf6815e020 0x7fbf681f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.883+0000 7fbf6d08c640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf681f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6d88d640 1 -- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 msgr2=0x7fbf681f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6d88d640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf681f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6d88d640 1 -- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf6815e020 msgr2=0x7fbf681f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6d88d640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf6815e020 0x7fbf681f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6d88d640 1 -- 172.21.15.6:0/2413633724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf68153980 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6d88d640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf680ced00 0x7fbf68153150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbf64000c00 tx=0x7fbf640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf5e7fc640 1 -- 172.21.15.6:0/2413633724 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64040070 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf5e7fc640 1 -- 172.21.15.6:0/2413633724 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf640399f0 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf68153c60 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf5e7fc640 1 -- 172.21.15.6:0/2413633724 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64035040 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.884+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf68154220 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.885+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6808afe0 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.885+0000 7fbf5e7fc640 1 -- 172.21.15.6:0/2413633724 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbf64038020 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.886+0000 7fbf5e7fc640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbf4c06bc00 0x7fbf4c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.886+0000 7fbf5e7fc640 1 -- 172.21.15.6:0/2413633724 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbf640ac620 con 0x7fbf680ced00 2024-03-18T14:42:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.886+0000 7fbf6d08c640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbf4c06bc00 0x7fbf4c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:21.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.886+0000 7fbf6d08c640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbf4c06bc00 0x7fbf4c06e0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf680cd520 tx=0x7fbf58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:21.890+0000 7fbf5e7fc640 1 -- 172.21.15.6:0/2413633724 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbf6407a6c0 con 0x7fbf680ced00 2024-03-18T14:42:22.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.077+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf680b2c50 con 0x7fbf680ced00 2024-03-18T14:42:22.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.078+0000 7fbf5e7fc640 1 -- 172.21.15.6:0/2413633724 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fbf6407e760 con 0x7fbf680ced00 2024-03-18T14:42:22.081 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:22.081 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbf4c06bc00 msgr2=0x7fbf4c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbf4c06bc00 0x7fbf4c06e0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf680cd520 tx=0x7fbf58009040 comp rx=0 tx=0).stop 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf680ced00 msgr2=0x7fbf68153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf680ced00 0x7fbf68153150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbf64000c00 tx=0x7fbf640046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 shutdown_connections 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbf4c06bc00 0x7fbf4c06e0c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf6815e020 0x7fbf681f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf680d3210 0x7fbf681f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 --2- 172.21.15.6:0/2413633724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf680ced00 0x7fbf68153150 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 >> 172.21.15.6:0/2413633724 conn(0x7fbf68072ce0 msgr2=0x7fbf680b8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:22.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 shutdown_connections 2024-03-18T14:42:22.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.080+0000 7fbf6faf1640 1 -- 172.21.15.6:0/2413633724 wait complete. 2024-03-18T14:42:22.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:22.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.197+0000 7ffb96b7e640 1 Processor -- start 2024-03-18T14:42:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.197+0000 7ffb96b7e640 1 -- start start 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb96b7e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb9009a350 0x7ffb900cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb96b7e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb900d0300 con 0x7ffb9015e150 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb96b7e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb900d05e0 con 0x7ffb900d13a0 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb96b7e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9015e530 con 0x7ffb9009a350 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb9511b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb9009a350 0x7ffb900cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb9491a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffb9015e150 0x7ffb9006d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55082/0 (socket says 172.21.15.6:55082) 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb9491a640 1 -- 172.21.15.6:0/3363017846 learned_addr learned my addr 172.21.15.6:0/3363017846 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb8ffff640 1 --1- 172.21.15.6:0/3363017846 >> v1:172.21.15.92:6789/0 conn(0x7ffb900d13a0 0x7ffb90099c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37826/0 (socket says 172.21.15.6:37826) 2024-03-18T14:42:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb9511b640 1 -- 172.21.15.6:0/3363017846 >> v1:172.21.15.92:6789/0 conn(0x7ffb900d13a0 legacy=0x7ffb90099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.198+0000 7ffb8f7fe640 1 -- 172.21.15.6:0/3363017846 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1775433278 0 0) 0x7ffb900d05e0 con 0x7ffb900d13a0 2024-03-18T14:42:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.199+0000 7ffb9511b640 1 -- 172.21.15.6:0/3363017846 >> v1:172.21.15.6:6789/0 conn(0x7ffb9015e150 legacy=0x7ffb9006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.199+0000 7ffb9511b640 1 -- 172.21.15.6:0/3363017846 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb900d08c0 con 0x7ffb9009a350 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.199+0000 7ffb9511b640 1 --2- 172.21.15.6:0/3363017846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb900cc1b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffb780097a0 tx=0x7ffb7802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9242a378625664 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.199+0000 7ffb8f7fe640 1 -- 172.21.15.6:0/3363017846 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78037070 con 0x7ffb9009a350 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.199+0000 7ffb8f7fe640 1 -- 172.21.15.6:0/3363017846 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb78030710 con 0x7ffb9009a350 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.199+0000 7ffb8f7fe640 1 -- 172.21.15.6:0/3363017846 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78030ae0 con 0x7ffb9009a350 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.200+0000 7ffb96b7e640 1 -- 172.21.15.6:0/3363017846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 msgr2=0x7ffb900cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.200+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/3363017846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb900cc1b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffb780097a0 tx=0x7ffb7802b710 comp rx=0 tx=0).stop 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.200+0000 7ffb96b7e640 1 -- 172.21.15.6:0/3363017846 shutdown_connections 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.200+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/3363017846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb900cc1b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.200+0000 7ffb96b7e640 1 -- 172.21.15.6:0/3363017846 >> 172.21.15.6:0/3363017846 conn(0x7ffb900b3650 msgr2=0x7ffb900b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.200+0000 7ffb96b7e640 1 -- 172.21.15.6:0/3363017846 shutdown_connections 2024-03-18T14:42:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.200+0000 7ffb96b7e640 1 -- 172.21.15.6:0/3363017846 wait complete. 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.201+0000 7ffb96b7e640 1 Processor -- start 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.201+0000 7ffb96b7e640 1 -- start start 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb96b7e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb9014f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb96b7e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb900d13a0 0x7ffb90070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb96b7e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb9015e150 0x7ffb900710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb96b7e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9006c650 con 0x7ffb9009a350 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb96b7e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffb900d0e80 con 0x7ffb900d13a0 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb96b7e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ffb9006c7d0 con 0x7ffb9015e150 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb9014f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb9014f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55796/0 (socket says 172.21.15.6:55796) 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 -- 172.21.15.6:0/1734438089 learned_addr learned my addr 172.21.15.6:0/1734438089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9511b640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb9015e150 0x7ffb900710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb8ffff640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb900d13a0 0x7ffb90070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 -- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb9015e150 msgr2=0x7ffb900710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb9015e150 0x7ffb900710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 -- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb900d13a0 msgr2=0x7ffb90070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb900d13a0 0x7ffb90070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.202+0000 7ffb9491a640 1 -- 172.21.15.6:0/1734438089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb900cc2c0 con 0x7ffb9009a350 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.203+0000 7ffb9491a640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb9014f1f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c7b0 tx=0x7ffb8800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.203+0000 7ffb8d7fa640 1 -- 172.21.15.6:0/1734438089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb88018070 con 0x7ffb9009a350 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.203+0000 7ffb8d7fa640 1 -- 172.21.15.6:0/1734438089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb880072d0 con 0x7ffb9009a350 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.203+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb900cc5a0 con 0x7ffb9009a350 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.203+0000 7ffb8d7fa640 1 -- 172.21.15.6:0/1734438089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb880047e0 con 0x7ffb9009a350 2024-03-18T14:42:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.203+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb900ccb60 con 0x7ffb9009a350 2024-03-18T14:42:22.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.204+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb900751b0 con 0x7ffb9009a350 2024-03-18T14:42:22.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.204+0000 7ffb8d7fa640 1 -- 172.21.15.6:0/1734438089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ffb8801b070 con 0x7ffb9009a350 2024-03-18T14:42:22.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.205+0000 7ffb8d7fa640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ffb7006bc00 0x7ffb7006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.205+0000 7ffb8d7fa640 1 -- 172.21.15.6:0/1734438089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ffb8808a120 con 0x7ffb9009a350 2024-03-18T14:42:22.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.205+0000 7ffb8ffff640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ffb7006bc00 0x7ffb7006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.205+0000 7ffb8ffff640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ffb7006bc00 0x7ffb7006e0c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffb900cd8a0 tx=0x7ffb80001630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.209+0000 7ffb8d7fa640 1 -- 172.21.15.6:0/1734438089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ffb88058310 con 0x7ffb9009a350 2024-03-18T14:42:22.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.390+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffb900cf7d0 con 0x7ffb9009a350 2024-03-18T14:42:22.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.391+0000 7ffb8d7fa640 1 -- 172.21.15.6:0/1734438089 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7ffb8805c3b0 con 0x7ffb9009a350 2024-03-18T14:42:22.393 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ffb7006bc00 msgr2=0x7ffb7006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ffb7006bc00 0x7ffb7006e0c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffb900cd8a0 tx=0x7ffb80001630 comp rx=0 tx=0).stop 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 msgr2=0x7ffb9014f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb9014f1f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c7b0 tx=0x7ffb8800cc70 comp rx=0 tx=0).stop 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 shutdown_connections 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ffb7006bc00 0x7ffb7006e0c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb9015e150 0x7ffb900710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb900d13a0 0x7ffb90070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 --2- 172.21.15.6:0/1734438089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb9009a350 0x7ffb9014f1f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 >> 172.21.15.6:0/1734438089 conn(0x7ffb900b3650 msgr2=0x7ffb90099c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 shutdown_connections 2024-03-18T14:42:22.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.393+0000 7ffb96b7e640 1 -- 172.21.15.6:0/1734438089 wait complete. 2024-03-18T14:42:22.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:22.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.508+0000 7f1ede3b3640 1 Processor -- start 2024-03-18T14:42:22.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.508+0000 7f1ede3b3640 1 -- start start 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ede3b3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ede3b3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ed80b5740 0x7f1ed80b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ede3b3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ed80d1e40 0x7f1ed80d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ede3b3640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed80b60e0 con 0x7f1ed80b5740 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ede3b3640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed815e3b0 con 0x7f1ed80d1e40 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ede3b3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed80b8470 con 0x7f1ed815dfd0 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ed80b5740 0x7f1ed80b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ed80b5740 0x7f1ed80b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55812/0 (socket says 172.21.15.6:55812) 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 -- 172.21.15.6:0/2784386981 learned_addr learned my addr 172.21.15.6:0/2784386981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed7fff640 1 --2- 172.21.15.6:0/2784386981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1edc950640 1 --2- 172.21.15.6:0/2784386981 >> v2:172.21.15.92:3300/0 conn(0x7f1ed80d1e40 0x7f1ed80d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 -- 172.21.15.6:0/2784386981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 msgr2=0x7f1ed80b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 --2- 172.21.15.6:0/2784386981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 -- 172.21.15.6:0/2784386981 >> v2:172.21.15.92:3300/0 conn(0x7f1ed80d1e40 msgr2=0x7f1ed80d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 --2- 172.21.15.6:0/2784386981 >> v2:172.21.15.92:3300/0 conn(0x7f1ed80d1e40 0x7f1ed80d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed77fe640 1 -- 172.21.15.6:0/2784386981 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed80b8750 con 0x7f1ed80b5740 2024-03-18T14:42:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.509+0000 7f1ed7fff640 1 --2- 172.21.15.6:0/2784386981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:22.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.510+0000 7f1ed77fe640 1 --2- 172.21.15.6:0/2784386981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80b5740 0x7f1ed80b5ba0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80b8ff0 tx=0x7f1ec802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a76557ff3f5f644 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.510+0000 7f1ed6ffd640 1 -- 172.21.15.6:0/2784386981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8035020 con 0x7f1ed80b5740 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.510+0000 7f1ed6ffd640 1 -- 172.21.15.6:0/2784386981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec8031870 con 0x7f1ed80b5740 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.510+0000 7f1ed6ffd640 1 -- 172.21.15.6:0/2784386981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8031c40 con 0x7f1ed80b5740 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 -- 172.21.15.6:0/2784386981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80b5740 msgr2=0x7f1ed80b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/2784386981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80b5740 0x7f1ed80b5ba0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80b8ff0 tx=0x7f1ec802b910 comp rx=0 tx=0).stop 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 -- 172.21.15.6:0/2784386981 shutdown_connections 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/2784386981 >> v2:172.21.15.92:3300/0 conn(0x7f1ed80d1e40 0x7f1ed80d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/2784386981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80b5740 0x7f1ed80b5ba0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/2784386981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 -- 172.21.15.6:0/2784386981 >> 172.21.15.6:0/2784386981 conn(0x7f1ed8072d10 msgr2=0x7f1ed8073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 -- 172.21.15.6:0/2784386981 shutdown_connections 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.511+0000 7f1ede3b3640 1 -- 172.21.15.6:0/2784386981 wait complete. 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.512+0000 7f1ede3b3640 1 Processor -- start 2024-03-18T14:42:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.512+0000 7f1ede3b3640 1 -- start start 2024-03-18T14:42:22.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ede3b3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed80b5740 0x7f1ed81f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ede3b3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80d1e40 0x7f1ed80b3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ede3b3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ede3b3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed80b9b90 con 0x7f1ed80d1e40 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ede3b3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed80b8d10 con 0x7f1ed80b5740 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ede3b3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed80d2810 con 0x7f1ed815dfd0 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80d1e40 0x7f1ed80b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80d1e40 0x7f1ed80b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55824/0 (socket says 172.21.15.6:55824) 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 -- 172.21.15.6:0/333887316 learned_addr learned my addr 172.21.15.6:0/333887316 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed7fff640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed80b5740 0x7f1ed81f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1edc950640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 -- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 msgr2=0x7f1ed80b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 -- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed80b5740 msgr2=0x7f1ed81f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed80b5740 0x7f1ed81f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 -- 172.21.15.6:0/333887316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed814e990 con 0x7f1ed80d1e40 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.513+0000 7f1ed77fe640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80d1e40 0x7f1ed80b3800 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1ec802b950 tx=0x7f1ec80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.514+0000 7f1ed4ff9640 1 -- 172.21.15.6:0/333887316 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8040070 con 0x7f1ed80d1e40 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.514+0000 7f1ed4ff9640 1 -- 172.21.15.6:0/333887316 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec8031e30 con 0x7f1ed80d1e40 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.514+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed815e7b0 con 0x7f1ed80d1e40 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.514+0000 7f1ed4ff9640 1 -- 172.21.15.6:0/333887316 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8035040 con 0x7f1ed80d1e40 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.514+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed815ed70 con 0x7f1ed80d1e40 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.515+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed80751b0 con 0x7f1ed80d1e40 2024-03-18T14:42:22.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.515+0000 7f1ed4ff9640 1 -- 172.21.15.6:0/333887316 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1ec8038020 con 0x7f1ed80d1e40 2024-03-18T14:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.515+0000 7f1ed4ff9640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ec406bbd0 0x7f1ec406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.516+0000 7f1ed4ff9640 1 -- 172.21.15.6:0/333887316 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1ec80acbf0 con 0x7f1ed80d1e40 2024-03-18T14:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.516+0000 7f1ed7fff640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ec406bbd0 0x7f1ec406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.516+0000 7f1ed7fff640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ec406bbd0 0x7f1ec406e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009560 tx=0x7f1ecc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.520+0000 7f1ed4ff9640 1 -- 172.21.15.6:0/333887316 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1ec807abf0 con 0x7f1ed80d1e40 2024-03-18T14:42:22.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.702+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1ed815e560 con 0x7f1ed80d1e40 2024-03-18T14:42:22.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.702+0000 7f1ed4ff9640 1 -- 172.21.15.6:0/333887316 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f1ec807ec90 con 0x7f1ed80d1e40 2024-03-18T14:42:22.705 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ec406bbd0 msgr2=0x7f1ec406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ec406bbd0 0x7f1ec406e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009560 tx=0x7f1ecc009190 comp rx=0 tx=0).stop 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80d1e40 msgr2=0x7f1ed80b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80d1e40 0x7f1ed80b3800 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1ec802b950 tx=0x7f1ec80046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 shutdown_connections 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ec406bbd0 0x7f1ec406e090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed815dfd0 0x7f1ed80b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed80d1e40 0x7f1ed80b3800 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 --2- 172.21.15.6:0/333887316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed80b5740 0x7f1ed81f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 >> 172.21.15.6:0/333887316 conn(0x7f1ed8072d10 msgr2=0x7f1ed80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:22.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 shutdown_connections 2024-03-18T14:42:22.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.704+0000 7f1ede3b3640 1 -- 172.21.15.6:0/333887316 wait complete. 2024-03-18T14:42:22.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:22.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.819+0000 7f0073b90640 1 Processor -- start 2024-03-18T14:42:22.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.819+0000 7f0073b90640 1 -- start start 2024-03-18T14:42:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f0073b90640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f006c15dfd0 0x7f006c0b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f0073b90640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f006c0b5740 0x7f006c0b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f0073b90640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f006c15e3b0 con 0x7f006c0d1e40 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f0073b90640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f006c0b8470 con 0x7f006c15dfd0 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f0073b90640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f006c0b8750 con 0x7f006c0b5740 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f007192c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f006c15dfd0 0x7f006c0b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f007192c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f006c15dfd0 0x7f006c0b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55830/0 (socket says 172.21.15.6:55830) 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f007192c640 1 -- 172.21.15.6:0/1627928302 learned_addr learned my addr 172.21.15.6:0/1627928302 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.820+0000 7f007112b640 1 --2- 172.21.15.6:0/1627928302 >> v2:172.21.15.92:3300/0 conn(0x7f006c0b5740 0x7f006c0b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.821+0000 7f007192c640 1 -- 172.21.15.6:0/1627928302 >> v2:172.21.15.92:3300/0 conn(0x7f006c0b5740 msgr2=0x7f006c0b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.821+0000 7f007192c640 1 --2- 172.21.15.6:0/1627928302 >> v2:172.21.15.92:3300/0 conn(0x7f006c0b5740 0x7f006c0b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.821+0000 7f007192c640 1 -- 172.21.15.6:0/1627928302 >> v1:172.21.15.92:6789/0 conn(0x7f006c0d1e40 legacy=0x7f006c0d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.821+0000 7f007192c640 1 -- 172.21.15.6:0/1627928302 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f006c0b8a30 con 0x7f006c15dfd0 2024-03-18T14:42:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.821+0000 7f007192c640 1 --2- 172.21.15.6:0/1627928302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c15dfd0 0x7f006c0b9650 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f006c0ce4a0 tx=0x7f006402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f5983105e27d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.821+0000 7f007092a640 1 -- 172.21.15.6:0/1627928302 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064034070 con 0x7f006c15dfd0 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f007092a640 1 -- 172.21.15.6:0/1627928302 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0064030860 con 0x7f006c15dfd0 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f007092a640 1 -- 172.21.15.6:0/1627928302 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064030c30 con 0x7f006c15dfd0 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 -- 172.21.15.6:0/1627928302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c15dfd0 msgr2=0x7f006c0b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 --2- 172.21.15.6:0/1627928302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c15dfd0 0x7f006c0b9650 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f006c0ce4a0 tx=0x7f006402b9a0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 -- 172.21.15.6:0/1627928302 shutdown_connections 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 --2- 172.21.15.6:0/1627928302 >> v2:172.21.15.92:3300/0 conn(0x7f006c0b5740 0x7f006c0b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 --2- 172.21.15.6:0/1627928302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c15dfd0 0x7f006c0b9650 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 -- 172.21.15.6:0/1627928302 >> 172.21.15.6:0/1627928302 conn(0x7f006c072d10 msgr2=0x7f006c073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 -- 172.21.15.6:0/1627928302 shutdown_connections 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.822+0000 7f0073b90640 1 -- 172.21.15.6:0/1627928302 wait complete. 2024-03-18T14:42:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.823+0000 7f0073b90640 1 Processor -- start 2024-03-18T14:42:22.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.823+0000 7f0073b90640 1 -- start start 2024-03-18T14:42:22.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f0073b90640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f006c0b5740 0x7f006c0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f0073b90640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c0d1e40 0x7f006c15fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f0073b90640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f006c15dfd0 0x7f006c160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f0073b90640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f006c0b7e00 con 0x7f006c0d1e40 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f0073b90640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f006c0b9b90 con 0x7f006c0b5740 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f0073b90640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f006c0b7f80 con 0x7f006c15dfd0 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c0d1e40 0x7f006c15fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c0d1e40 0x7f006c15fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55844/0 (socket says 172.21.15.6:55844) 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 -- 172.21.15.6:0/735889941 learned_addr learned my addr 172.21.15.6:0/735889941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007192c640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f006c0b5740 0x7f006c0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007212d640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f006c15dfd0 0x7f006c160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 -- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f006c15dfd0 msgr2=0x7f006c160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f006c15dfd0 0x7f006c160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 -- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f006c0b5740 msgr2=0x7f006c0b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f006c0b5740 0x7f006c0b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007112b640 1 -- 172.21.15.6:0/735889941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f006c0c8600 con 0x7f006c0d1e40 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.824+0000 7f007212d640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f006c15dfd0 0x7f006c160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.825+0000 7f007112b640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c0d1e40 0x7f006c15fdc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f006000e9b0 tx=0x7f006000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.825+0000 7f006a7fc640 1 -- 172.21.15.6:0/735889941 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060017070 con 0x7f006c0d1e40 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.825+0000 7f006a7fc640 1 -- 172.21.15.6:0/735889941 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f006000c9a0 con 0x7f006c0d1e40 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.825+0000 7f006a7fc640 1 -- 172.21.15.6:0/735889941 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006000cd90 con 0x7f006c0d1e40 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.825+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f006c0c88e0 con 0x7f006c0d1e40 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.825+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f006c0c8ea0 con 0x7f006c0d1e40 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.826+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f006c0751b0 con 0x7f006c0d1e40 2024-03-18T14:42:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.826+0000 7f006a7fc640 1 -- 172.21.15.6:0/735889941 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f006001a070 con 0x7f006c0d1e40 2024-03-18T14:42:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.827+0000 7f006a7fc640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f005806bc00 0x7f005806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.827+0000 7f006a7fc640 1 -- 172.21.15.6:0/735889941 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f006008b150 con 0x7f006c0d1e40 2024-03-18T14:42:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.827+0000 7f007192c640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f005806bc00 0x7f005806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.827+0000 7f007192c640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f005806bc00 0x7f005806e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0064036000 tx=0x7f006403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:22.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:22.831+0000 7f006a7fc640 1 -- 172.21.15.6:0/735889941 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f00600591f0 con 0x7f006c0d1e40 2024-03-18T14:42:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.012+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f006c0c6de0 con 0x7f006c0d1e40 2024-03-18T14:42:23.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.012+0000 7f006a7fc640 1 -- 172.21.15.6:0/735889941 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f006005f030 con 0x7f006c0d1e40 2024-03-18T14:42:23.015 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f005806bc00 msgr2=0x7f005806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f005806bc00 0x7f005806e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0064036000 tx=0x7f006403b040 comp rx=0 tx=0).stop 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c0d1e40 msgr2=0x7f006c15fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c0d1e40 0x7f006c15fdc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f006000e9b0 tx=0x7f006000ee70 comp rx=0 tx=0).stop 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 shutdown_connections 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f005806bc00 0x7f005806e0c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f006c15dfd0 0x7f006c160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f006c0d1e40 0x7f006c15fdc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 --2- 172.21.15.6:0/735889941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f006c0b5740 0x7f006c0b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 >> 172.21.15.6:0/735889941 conn(0x7f006c072d10 msgr2=0x7f006c0b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 shutdown_connections 2024-03-18T14:42:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.014+0000 7f0073b90640 1 -- 172.21.15.6:0/735889941 wait complete. 2024-03-18T14:42:23.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:23.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.129+0000 7f0117dd4640 1 Processor -- start 2024-03-18T14:42:23.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.129+0000 7f0117dd4640 1 -- start start 2024-03-18T14:42:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.129+0000 7f0117dd4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f011015dfd0 0x7f01100b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.129+0000 7f0117dd4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100b5740 0x7f01100b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.129+0000 7f0117dd4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f01100d1e40 0x7f01100d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.129+0000 7f0117dd4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f01100b60e0 con 0x7f011015dfd0 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0117dd4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f011015e3b0 con 0x7f01100d1e40 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0117dd4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f01100b8470 con 0x7f01100b5740 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f011015dfd0 0x7f01100b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f011015dfd0 0x7f01100b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55852/0 (socket says 172.21.15.6:55852) 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 -- 172.21.15.6:0/183854490 learned_addr learned my addr 172.21.15.6:0/183854490 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0116371640 1 --2- 172.21.15.6:0/183854490 >> v2:172.21.15.92:3300/0 conn(0x7f01100d1e40 0x7f01100d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f011536f640 1 --2- 172.21.15.6:0/183854490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100b5740 0x7f01100b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 -- 172.21.15.6:0/183854490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100b5740 msgr2=0x7f01100b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 --2- 172.21.15.6:0/183854490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100b5740 0x7f01100b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 -- 172.21.15.6:0/183854490 >> v2:172.21.15.92:3300/0 conn(0x7f01100d1e40 msgr2=0x7f01100d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:23.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 --2- 172.21.15.6:0/183854490 >> v2:172.21.15.92:3300/0 conn(0x7f01100d1e40 0x7f01100d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0115b70640 1 -- 172.21.15.6:0/183854490 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01100b8750 con 0x7f011015dfd0 2024-03-18T14:42:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.130+0000 7f0116371640 1 --2- 172.21.15.6:0/183854490 >> v2:172.21.15.92:3300/0 conn(0x7f01100d1e40 0x7f01100d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.131+0000 7f0115b70640 1 --2- 172.21.15.6:0/183854490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f011015dfd0 0x7f01100b9650 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f01100b8ff0 tx=0x7f010c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86ae4a521e9e4c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.131+0000 7f0114b6e640 1 -- 172.21.15.6:0/183854490 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c035020 con 0x7f011015dfd0 2024-03-18T14:42:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.131+0000 7f0114b6e640 1 -- 172.21.15.6:0/183854490 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f010c031870 con 0x7f011015dfd0 2024-03-18T14:42:23.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.131+0000 7f0114b6e640 1 -- 172.21.15.6:0/183854490 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c031c40 con 0x7f011015dfd0 2024-03-18T14:42:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.131+0000 7f0117dd4640 1 -- 172.21.15.6:0/183854490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f011015dfd0 msgr2=0x7f01100b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.131+0000 7f0117dd4640 1 --2- 172.21.15.6:0/183854490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f011015dfd0 0x7f01100b9650 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f01100b8ff0 tx=0x7f010c02b910 comp rx=0 tx=0).stop 2024-03-18T14:42:23.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.132+0000 7f0117dd4640 1 -- 172.21.15.6:0/183854490 shutdown_connections 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.132+0000 7f0117dd4640 1 --2- 172.21.15.6:0/183854490 >> v2:172.21.15.92:3300/0 conn(0x7f01100d1e40 0x7f01100d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.132+0000 7f0117dd4640 1 --2- 172.21.15.6:0/183854490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100b5740 0x7f01100b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.132+0000 7f0117dd4640 1 --2- 172.21.15.6:0/183854490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f011015dfd0 0x7f01100b9650 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.132+0000 7f0117dd4640 1 -- 172.21.15.6:0/183854490 >> 172.21.15.6:0/183854490 conn(0x7f0110072d10 msgr2=0x7f0110073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.132+0000 7f0117dd4640 1 -- 172.21.15.6:0/183854490 shutdown_connections 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.132+0000 7f0117dd4640 1 -- 172.21.15.6:0/183854490 wait complete. 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 Processor -- start 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 -- start start 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01100b5740 0x7f011014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100d1e40 0x7f01100b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f011015dfd0 0x7f01100b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f01100b9b90 con 0x7f01100b5740 2024-03-18T14:42:23.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f01100b8d10 con 0x7f011015dfd0 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.133+0000 7f0117dd4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f01100d2810 con 0x7f01100d1e40 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01100b5740 0x7f011014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01100b5740 0x7f011014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55866/0 (socket says 172.21.15.6:55866) 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 -- 172.21.15.6:0/1379206066 learned_addr learned my addr 172.21.15.6:0/1379206066 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0116371640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f011015dfd0 0x7f01100b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f011536f640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100d1e40 0x7f01100b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 -- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100d1e40 msgr2=0x7f01100b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100d1e40 0x7f01100b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 -- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f011015dfd0 msgr2=0x7f01100b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f011015dfd0 0x7f01100b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 -- 172.21.15.6:0/1379206066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011014f520 con 0x7f01100b5740 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0116371640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f011015dfd0 0x7f01100b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:23.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0115b70640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01100b5740 0x7f011014efe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f010c02b950 tx=0x7f010c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f01067fc640 1 -- 172.21.15.6:0/1379206066 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c040070 con 0x7f01100b5740 2024-03-18T14:42:23.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f01067fc640 1 -- 172.21.15.6:0/1379206066 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f010c031e10 con 0x7f01100b5740 2024-03-18T14:42:23.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011015f220 con 0x7f01100b5740 2024-03-18T14:42:23.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f01067fc640 1 -- 172.21.15.6:0/1379206066 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c035040 con 0x7f01100b5740 2024-03-18T14:42:23.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.134+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0110160a50 con 0x7f01100b5740 2024-03-18T14:42:23.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.135+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01100751b0 con 0x7f01100b5740 2024-03-18T14:42:23.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.136+0000 7f01067fc640 1 -- 172.21.15.6:0/1379206066 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f010c04b060 con 0x7f01100b5740 2024-03-18T14:42:23.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.136+0000 7f01067fc640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f00f406bbd0 0x7f00f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.136+0000 7f01067fc640 1 -- 172.21.15.6:0/1379206066 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f010c0ac890 con 0x7f01100b5740 2024-03-18T14:42:23.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.136+0000 7f011536f640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f00f406bbd0 0x7f00f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.137+0000 7f011536f640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f00f406bbd0 0x7f00f406e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f01100b5e80 tx=0x7f0100009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.140+0000 7f01067fc640 1 -- 172.21.15.6:0/1379206066 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f010c07b010 con 0x7f01100b5740 2024-03-18T14:42:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.321+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f01100cdba0 con 0x7f01100b5740 2024-03-18T14:42:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.321+0000 7f01067fc640 1 -- 172.21.15.6:0/1379206066 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f010c07a910 con 0x7f01100b5740 2024-03-18T14:42:23.324 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f00f406bbd0 msgr2=0x7f00f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f00f406bbd0 0x7f00f406e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f01100b5e80 tx=0x7f0100009190 comp rx=0 tx=0).stop 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01100b5740 msgr2=0x7f011014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01100b5740 0x7f011014efe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f010c02b950 tx=0x7f010c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 shutdown_connections 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f00f406bbd0 0x7f00f406e090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f011015dfd0 0x7f01100b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f01100d1e40 0x7f01100b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 --2- 172.21.15.6:0/1379206066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01100b5740 0x7f011014efe0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 >> 172.21.15.6:0/1379206066 conn(0x7f0110072d10 msgr2=0x7f01100b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 shutdown_connections 2024-03-18T14:42:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.323+0000 7f0117dd4640 1 -- 172.21.15.6:0/1379206066 wait complete. 2024-03-18T14:42:23.341 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2024-03-18T14:42:23.341 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-18T14:42:23.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:23.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.441+0000 7fa3e624b640 1 Processor -- start 2024-03-18T14:42:23.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.441+0000 7fa3e624b640 1 -- start start 2024-03-18T14:42:23.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3e624b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3e00cd8c0 0x7fa3e00b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3e624b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e00b27e0 con 0x7fa3e00b23c0 2024-03-18T14:42:23.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3e624b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e00b0c60 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3e624b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa3e00b0f40 con 0x7fa3e00cd8c0 2024-03-18T14:42:23.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3deffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3e00cd8c0 0x7fa3e00b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3dffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3e00b23c0 0x7fa3e00cca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55096/0 (socket says 172.21.15.6:55096) 2024-03-18T14:42:23.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3df7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa3e00cf9d0 0x7fa3e00d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37856/0 (socket says 172.21.15.6:37856) 2024-03-18T14:42:23.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3dffff640 1 -- 172.21.15.6:0/2250119316 learned_addr learned my addr 172.21.15.6:0/2250119316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:23.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3deffd640 1 -- 172.21.15.6:0/2250119316 >> v1:172.21.15.92:6789/0 conn(0x7fa3e00cf9d0 legacy=0x7fa3e00d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3deffd640 1 -- 172.21.15.6:0/2250119316 >> v1:172.21.15.6:6789/0 conn(0x7fa3e00b23c0 legacy=0x7fa3e00cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3de7fc640 1 -- 172.21.15.6:0/2250119316 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011212656 0 0) 0x7fa3e00b0c60 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.442+0000 7fa3deffd640 1 -- 172.21.15.6:0/2250119316 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e00b1220 con 0x7fa3e00cd8c0 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.443+0000 7fa3deffd640 1 --2- 172.21.15.6:0/2250119316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cd8c0 0x7fa3e00b1e80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40097a0 tx=0x7fa3d402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b1a8995ca4bfd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.443+0000 7fa3de7fc640 1 -- 172.21.15.6:0/2250119316 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4037070 con 0x7fa3e00cd8c0 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.443+0000 7fa3de7fc640 1 -- 172.21.15.6:0/2250119316 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3d4030710 con 0x7fa3e00cd8c0 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.443+0000 7fa3de7fc640 1 -- 172.21.15.6:0/2250119316 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4030ae0 con 0x7fa3e00cd8c0 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.444+0000 7fa3e624b640 1 -- 172.21.15.6:0/2250119316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cd8c0 msgr2=0x7fa3e00b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.444+0000 7fa3e624b640 1 --2- 172.21.15.6:0/2250119316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cd8c0 0x7fa3e00b1e80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40097a0 tx=0x7fa3d402b790 comp rx=0 tx=0).stop 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.444+0000 7fa3e624b640 1 -- 172.21.15.6:0/2250119316 shutdown_connections 2024-03-18T14:42:23.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.444+0000 7fa3e624b640 1 --2- 172.21.15.6:0/2250119316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cd8c0 0x7fa3e00b1e80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.444+0000 7fa3e624b640 1 -- 172.21.15.6:0/2250119316 >> 172.21.15.6:0/2250119316 conn(0x7fa3e00b5340 msgr2=0x7fa3e00b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:23.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.444+0000 7fa3e624b640 1 -- 172.21.15.6:0/2250119316 shutdown_connections 2024-03-18T14:42:23.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.444+0000 7fa3e624b640 1 -- 172.21.15.6:0/2250119316 wait complete. 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.445+0000 7fa3e624b640 1 Processor -- start 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.445+0000 7fa3e624b640 1 -- start start 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3e624b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3e00b23c0 0x7fa3e006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3e624b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3e00cd8c0 0x7fa3e00c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3e624b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cf9d0 0x7fa3e015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3e624b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e00cfdb0 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3e624b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e00b17e0 con 0x7fa3e00cd8c0 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3e624b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa3e00cff30 con 0x7fa3e00b23c0 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cf9d0 0x7fa3e015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cf9d0 0x7fa3e015b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55884/0 (socket says 172.21.15.6:55884) 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 -- 172.21.15.6:0/1110008922 learned_addr learned my addr 172.21.15.6:0/1110008922 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:23.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3df7fe640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3e00b23c0 0x7fa3e006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3deffd640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3e00cd8c0 0x7fa3e00c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 -- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3e00b23c0 msgr2=0x7fa3e006faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3e00b23c0 0x7fa3e006faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 -- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3e00cd8c0 msgr2=0x7fa3e00c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3e00cd8c0 0x7fa3e00c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.446+0000 7fa3dffff640 1 -- 172.21.15.6:0/1110008922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e015e6e0 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.447+0000 7fa3dffff640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cf9d0 0x7fa3e015b5f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800caf0 tx=0x7fa3d8002090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.447+0000 7fa3c7fff640 1 -- 172.21.15.6:0/1110008922 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8018070 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.447+0000 7fa3c7fff640 1 -- 172.21.15.6:0/1110008922 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3d800a700 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.447+0000 7fa3c7fff640 1 -- 172.21.15.6:0/1110008922 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d800aaf0 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.447+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e015e9c0 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.447+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e015ef80 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.448+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e009e550 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.448+0000 7fa3c7fff640 1 -- 172.21.15.6:0/1110008922 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3d801b070 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.449+0000 7fa3c7fff640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.449+0000 7fa3c7fff640 1 -- 172.21.15.6:0/1110008922 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa3d808a560 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.449+0000 7fa3df7fe640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.449+0000 7fa3df7fe640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007fd0 tx=0x7fa3d0007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.453+0000 7fa3c7fff640 1 -- 172.21.15.6:0/1110008922 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3d8058600 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.638+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3e00b59c0 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.639+0000 7fa3c7fff640 1 -- 172.21.15.6:0/1110008922 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fa3d805c6a0 con 0x7fa3e00cf9d0 2024-03-18T14:42:23.642 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:23.642 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 msgr2=0x7fa3cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007fd0 tx=0x7fa3d0007de0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cf9d0 msgr2=0x7fa3e015b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cf9d0 0x7fa3e015b5f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800caf0 tx=0x7fa3d8002090 comp rx=0 tx=0).stop 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 shutdown_connections 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3cc06bbd0 0x7fa3cc06e090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3e00cf9d0 0x7fa3e015b5f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3e00cd8c0 0x7fa3e00c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 --2- 172.21.15.6:0/1110008922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3e00b23c0 0x7fa3e006faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 >> 172.21.15.6:0/1110008922 conn(0x7fa3e00b5340 msgr2=0x7fa3e006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:23.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 shutdown_connections 2024-03-18T14:42:23.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.641+0000 7fa3e624b640 1 -- 172.21.15.6:0/1110008922 wait complete. 2024-03-18T14:42:23.655 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:23.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:23.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.759+0000 7f0b5de85640 1 Processor -- start 2024-03-18T14:42:23.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.759+0000 7f0b5de85640 1 -- start start 2024-03-18T14:42:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b5de85640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b580d3640 0x7f0b58168a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b5de85640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b5815eb20 con 0x7f0b580ce970 2024-03-18T14:42:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b5de85640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b5815edd0 con 0x7f0b580b9940 2024-03-18T14:42:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b5de85640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0b580d21a0 con 0x7f0b580d3640 2024-03-18T14:42:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b57fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b580d3640 0x7f0b58168a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b577fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b580ce970 0x7f0b580ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55866/0 (socket says 172.21.15.6:55866) 2024-03-18T14:42:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b577fe640 1 -- 172.21.15.6:0/3712564203 learned_addr learned my addr 172.21.15.6:0/3712564203 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b567fc640 1 -- 172.21.15.6:0/3712564203 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225232425 0 0) 0x7f0b5815eb20 con 0x7f0b580ce970 2024-03-18T14:42:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b567fc640 1 -- 172.21.15.6:0/3712564203 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b40003520 con 0x7f0b580ce970 2024-03-18T14:42:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b567fc640 1 -- 172.21.15.6:0/3712564203 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1960117828 0 0) 0x7f0b5815edd0 con 0x7f0b580b9940 2024-03-18T14:42:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b57fff640 1 -- 172.21.15.6:0/3712564203 >> v1:172.21.15.92:6789/0 conn(0x7f0b580b9940 legacy=0x7f0b5815e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b57fff640 1 -- 172.21.15.6:0/3712564203 >> v1:172.21.15.6:6789/0 conn(0x7f0b580ce970 legacy=0x7f0b580ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.760+0000 7f0b57fff640 1 -- 172.21.15.6:0/3712564203 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5815f0b0 con 0x7f0b580d3640 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.761+0000 7f0b57fff640 1 --2- 172.21.15.6:0/3712564203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b58168a90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009d90 tx=0x7f0b5002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72e79897ba3eb75 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.761+0000 7f0b567fc640 1 -- 172.21.15.6:0/3712564203 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50039070 con 0x7f0b580d3640 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.761+0000 7f0b567fc640 1 -- 172.21.15.6:0/3712564203 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5002ccf0 con 0x7f0b580d3640 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.761+0000 7f0b567fc640 1 -- 172.21.15.6:0/3712564203 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50034660 con 0x7f0b580d3640 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.762+0000 7f0b5de85640 1 -- 172.21.15.6:0/3712564203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 msgr2=0x7f0b58168a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.762+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3712564203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b58168a90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009d90 tx=0x7f0b5002bf40 comp rx=0 tx=0).stop 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.762+0000 7f0b5de85640 1 -- 172.21.15.6:0/3712564203 shutdown_connections 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.762+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3712564203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b58168a90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.762+0000 7f0b5de85640 1 -- 172.21.15.6:0/3712564203 >> 172.21.15.6:0/3712564203 conn(0x7f0b5806c900 msgr2=0x7f0b5806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.762+0000 7f0b5de85640 1 -- 172.21.15.6:0/3712564203 shutdown_connections 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.762+0000 7f0b5de85640 1 -- 172.21.15.6:0/3712564203 wait complete. 2024-03-18T14:42:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.763+0000 7f0b5de85640 1 Processor -- start 2024-03-18T14:42:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.763+0000 7f0b5de85640 1 -- start start 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b5de85640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b580b9940 0x7f0b58155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b5de85640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b580ce970 0x7f0b580d3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b5de85640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b580d41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b5de85640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5815bd70 con 0x7f0b580d3640 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b5de85640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5815f390 con 0x7f0b580b9940 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b5de85640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0b580cade0 con 0x7f0b580ce970 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b580d41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b580d41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36756/0 (socket says 172.21.15.6:36756) 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 -- 172.21.15.6:0/3244301858 learned_addr learned my addr 172.21.15.6:0/3244301858 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b577fe640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b580b9940 0x7f0b58155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b56ffd640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b580ce970 0x7f0b580d3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 -- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b580ce970 msgr2=0x7f0b580d3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b580ce970 0x7f0b580d3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 -- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b580b9940 msgr2=0x7f0b58155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b580b9940 0x7f0b58155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 -- 172.21.15.6:0/3244301858 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b580c9730 con 0x7f0b580d3640 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.764+0000 7f0b57fff640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b580d41c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009d90 tx=0x7f0b50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.765+0000 7f0b3ffff640 1 -- 172.21.15.6:0/3244301858 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50039050 con 0x7f0b580d3640 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.765+0000 7f0b3ffff640 1 -- 172.21.15.6:0/3244301858 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5003a040 con 0x7f0b580d3640 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.765+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b580b38a0 con 0x7f0b580d3640 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.765+0000 7f0b3ffff640 1 -- 172.21.15.6:0/3244301858 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50009450 con 0x7f0b580d3640 2024-03-18T14:42:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.765+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b58155f70 con 0x7f0b580d3640 2024-03-18T14:42:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.766+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b580751b0 con 0x7f0b580d3640 2024-03-18T14:42:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.766+0000 7f0b3ffff640 1 -- 172.21.15.6:0/3244301858 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0b50047020 con 0x7f0b580d3640 2024-03-18T14:42:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.766+0000 7f0b3ffff640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b4406bbd0 0x7f0b4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.767+0000 7f0b3ffff640 1 -- 172.21.15.6:0/3244301858 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0b500acf50 con 0x7f0b580d3640 2024-03-18T14:42:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.767+0000 7f0b577fe640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b4406bbd0 0x7f0b4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.767+0000 7f0b577fe640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b4406bbd0 0x7f0b4406e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0b48005880 tx=0x7f0b48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:23.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.771+0000 7f0b3ffff640 1 -- 172.21.15.6:0/3244301858 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0b5004e020 con 0x7f0b580d3640 2024-03-18T14:42:23.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.954+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0b5806b8a0 con 0x7f0b580d3640 2024-03-18T14:42:23.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.954+0000 7f0b3ffff640 1 -- 172.21.15.6:0/3244301858 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f0b5007b080 con 0x7f0b580d3640 2024-03-18T14:42:23.957 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b4406bbd0 msgr2=0x7f0b4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b4406bbd0 0x7f0b4406e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0b48005880 tx=0x7f0b48009040 comp rx=0 tx=0).stop 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 msgr2=0x7f0b580d41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b580d41c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009d90 tx=0x7f0b50004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 shutdown_connections 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b4406bbd0 0x7f0b4406e090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b580d3640 0x7f0b580d41c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b580ce970 0x7f0b580d3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 --2- 172.21.15.6:0/3244301858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b580b9940 0x7f0b58155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:23.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 >> 172.21.15.6:0/3244301858 conn(0x7f0b5806c900 msgr2=0x7f0b5815ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:23.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 shutdown_connections 2024-03-18T14:42:23.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:23.957+0000 7f0b5de85640 1 -- 172.21.15.6:0/3244301858 wait complete. 2024-03-18T14:42:23.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.072+0000 7f5b17644640 1 Processor -- start 2024-03-18T14:42:24.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.072+0000 7f5b17644640 1 -- start start 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b17644640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b10099c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b17644640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b100d0300 con 0x7f5b1009a180 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b17644640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b100d05e0 con 0x7f5b1015e150 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b17644640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1015e530 con 0x7f5b100d13a0 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b15be1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5b1009a180 0x7f5b100cc1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55882/0 (socket says 172.21.15.6:55882) 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b15be1640 1 -- 172.21.15.6:0/2200944058 learned_addr learned my addr 172.21.15.6:0/2200944058 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b14bdf640 1 --2- 172.21.15.6:0/2200944058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b10099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b07fff640 1 -- 172.21.15.6:0/2200944058 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200975560 0 0) 0x7f5b100d0300 con 0x7f5b1009a180 2024-03-18T14:42:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b07fff640 1 -- 172.21.15.6:0/2200944058 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5afc003520 con 0x7f5b1009a180 2024-03-18T14:42:24.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b07fff640 1 -- 172.21.15.6:0/2200944058 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1357595754 0 0) 0x7f5b100d05e0 con 0x7f5b1015e150 2024-03-18T14:42:24.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b07fff640 1 -- 172.21.15.6:0/2200944058 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b100d0300 con 0x7f5b1015e150 2024-03-18T14:42:24.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b14bdf640 1 -- 172.21.15.6:0/2200944058 >> v1:172.21.15.92:6789/0 conn(0x7f5b1015e150 legacy=0x7f5b1006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b14bdf640 1 -- 172.21.15.6:0/2200944058 >> v1:172.21.15.6:6789/0 conn(0x7f5b1009a180 legacy=0x7f5b100cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.073+0000 7f5b14bdf640 1 -- 172.21.15.6:0/2200944058 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b100d08c0 con 0x7f5b100d13a0 2024-03-18T14:42:24.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.074+0000 7f5b14bdf640 1 --2- 172.21.15.6:0/2200944058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b10099c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b000097d0 tx=0x7f5b0002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce0fd5b52af2c92d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.074+0000 7f5b07fff640 1 -- 172.21.15.6:0/2200944058 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00035020 con 0x7f5b100d13a0 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.074+0000 7f5b07fff640 1 -- 172.21.15.6:0/2200944058 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00031800 con 0x7f5b100d13a0 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.074+0000 7f5b07fff640 1 -- 172.21.15.6:0/2200944058 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00031bb0 con 0x7f5b100d13a0 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.075+0000 7f5b17644640 1 -- 172.21.15.6:0/2200944058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 msgr2=0x7f5b10099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.075+0000 7f5b17644640 1 --2- 172.21.15.6:0/2200944058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b10099c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b000097d0 tx=0x7f5b0002b8a0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.075+0000 7f5b17644640 1 -- 172.21.15.6:0/2200944058 shutdown_connections 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.075+0000 7f5b17644640 1 --2- 172.21.15.6:0/2200944058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b10099c40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.075+0000 7f5b17644640 1 -- 172.21.15.6:0/2200944058 >> 172.21.15.6:0/2200944058 conn(0x7f5b100b3650 msgr2=0x7f5b100b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.075+0000 7f5b17644640 1 -- 172.21.15.6:0/2200944058 shutdown_connections 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.075+0000 7f5b17644640 1 -- 172.21.15.6:0/2200944058 wait complete. 2024-03-18T14:42:24.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.076+0000 7f5b17644640 1 Processor -- start 2024-03-18T14:42:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.076+0000 7f5b17644640 1 -- start start 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b17644640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b1009a180 0x7f5b100b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b17644640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b100b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b17644640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b1015e150 0x7f5b10154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b17644640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1009a5d0 con 0x7f5b1009a180 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b17644640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5b100d17e0 con 0x7f5b1015e150 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b17644640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5b100d0e80 con 0x7f5b100d13a0 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b1009a180 0x7f5b100b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b1009a180 0x7f5b100b4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36772/0 (socket says 172.21.15.6:36772) 2024-03-18T14:42:24.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 -- 172.21.15.6:0/3220923524 learned_addr learned my addr 172.21.15.6:0/3220923524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b15be1640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b1015e150 0x7f5b10154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b14bdf640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b100b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 -- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 msgr2=0x7f5b100b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b100b6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 -- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b1015e150 msgr2=0x7f5b10154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b1015e150 0x7f5b10154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 -- 172.21.15.6:0/3220923524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b100b5030 con 0x7f5b1009a180 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b15be1640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b1015e150 0x7f5b10154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b153e0640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b1009a180 0x7f5b100b4750 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00c460 tx=0x7f5b0c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b05ffb640 1 -- 172.21.15.6:0/3220923524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c017070 con 0x7f5b1009a180 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b05ffb640 1 -- 172.21.15.6:0/3220923524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0c0052f0 con 0x7f5b1009a180 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b100b5310 con 0x7f5b1009a180 2024-03-18T14:42:24.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.077+0000 7f5b05ffb640 1 -- 172.21.15.6:0/3220923524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c003830 con 0x7f5b1009a180 2024-03-18T14:42:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.078+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b100b58d0 con 0x7f5b1009a180 2024-03-18T14:42:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.079+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b100751b0 con 0x7f5b1009a180 2024-03-18T14:42:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.079+0000 7f5b05ffb640 1 -- 172.21.15.6:0/3220923524 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5b0c01a070 con 0x7f5b1009a180 2024-03-18T14:42:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.079+0000 7f5b05ffb640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5af406bbd0 0x7f5af406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.080+0000 7f5b05ffb640 1 -- 172.21.15.6:0/3220923524 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5b0c08a140 con 0x7f5b1009a180 2024-03-18T14:42:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.080+0000 7f5b14bdf640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5af406bbd0 0x7f5af406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.080+0000 7f5b14bdf640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5af406bbd0 0x7f5af406e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5b00036000 tx=0x7f5b0003c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.084+0000 7f5b05ffb640 1 -- 172.21.15.6:0/3220923524 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5b0c0581e0 con 0x7f5b1009a180 2024-03-18T14:42:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.266+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5b10071b50 con 0x7f5b1009a180 2024-03-18T14:42:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.266+0000 7f5b05ffb640 1 -- 172.21.15.6:0/3220923524 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f5b0c05c280 con 0x7f5b1009a180 2024-03-18T14:42:24.269 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:24.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.268+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5af406bbd0 msgr2=0x7f5af406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.268+0000 7f5b17644640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5af406bbd0 0x7f5af406e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5b00036000 tx=0x7f5b0003c040 comp rx=0 tx=0).stop 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.268+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b1009a180 msgr2=0x7f5b100b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.268+0000 7f5b17644640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b1009a180 0x7f5b100b4750 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00c460 tx=0x7f5b0c00c920 comp rx=0 tx=0).stop 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.268+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 shutdown_connections 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.269+0000 7f5b17644640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5af406bbd0 0x7f5af406e090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.269+0000 7f5b17644640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b1015e150 0x7f5b10154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.269+0000 7f5b17644640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b100d13a0 0x7f5b100b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.269+0000 7f5b17644640 1 --2- 172.21.15.6:0/3220923524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b1009a180 0x7f5b100b4750 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.269+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 >> 172.21.15.6:0/3220923524 conn(0x7f5b100b3650 msgr2=0x7f5b100afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.269+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 shutdown_connections 2024-03-18T14:42:24.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.269+0000 7f5b17644640 1 -- 172.21.15.6:0/3220923524 wait complete. 2024-03-18T14:42:24.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:24.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.384+0000 7f489d0e8640 1 Processor -- start 2024-03-18T14:42:24.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.384+0000 7f489d0e8640 1 -- start start 2024-03-18T14:42:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489d0e8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f489815cf10 0x7f48980af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489d0e8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48980afef0 con 0x7f489815dac0 2024-03-18T14:42:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489d0e8640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48980b01d0 con 0x7f4898071ad0 2024-03-18T14:42:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489d0e8640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f489815d370 con 0x7f489815cf10 2024-03-18T14:42:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489759e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f489815dac0 0x7f489815df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55894/0 (socket says 172.21.15.6:55894) 2024-03-18T14:42:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489759e640 1 -- 172.21.15.6:0/1616854510 learned_addr learned my addr 172.21.15.6:0/1616854510 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f488ffff640 1 --2- 172.21.15.6:0/1616854510 >> v2:172.21.15.92:3300/0 conn(0x7f489815cf10 0x7f48980af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489659c640 1 -- 172.21.15.6:0/1616854510 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519530046 0 0) 0x7f48980afef0 con 0x7f489815dac0 2024-03-18T14:42:24.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489659c640 1 -- 172.21.15.6:0/1616854510 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4884003520 con 0x7f489815dac0 2024-03-18T14:42:24.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489659c640 1 -- 172.21.15.6:0/1616854510 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3217458378 0 0) 0x7f48980b01d0 con 0x7f4898071ad0 2024-03-18T14:42:24.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f489659c640 1 -- 172.21.15.6:0/1616854510 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48980afef0 con 0x7f4898071ad0 2024-03-18T14:42:24.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f488ffff640 1 -- 172.21.15.6:0/1616854510 >> v1:172.21.15.92:6789/0 conn(0x7f4898071ad0 legacy=0x7f4898071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f488ffff640 1 -- 172.21.15.6:0/1616854510 >> v1:172.21.15.6:6789/0 conn(0x7f489815dac0 legacy=0x7f489815df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.385+0000 7f488ffff640 1 -- 172.21.15.6:0/1616854510 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48980b04b0 con 0x7f489815cf10 2024-03-18T14:42:24.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.386+0000 7f488ffff640 1 --2- 172.21.15.6:0/1616854510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 0x7f48980af580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f48800097a0 tx=0x7f488002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1321973cb81a43e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.386+0000 7f489659c640 1 -- 172.21.15.6:0/1616854510 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880034070 con 0x7f489815cf10 2024-03-18T14:42:24.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.386+0000 7f489659c640 1 -- 172.21.15.6:0/1616854510 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4880030610 con 0x7f489815cf10 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.386+0000 7f489659c640 1 -- 172.21.15.6:0/1616854510 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48800309c0 con 0x7f489815cf10 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.387+0000 7f489d0e8640 1 -- 172.21.15.6:0/1616854510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 msgr2=0x7f48980af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.387+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1616854510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 0x7f48980af580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f48800097a0 tx=0x7f488002b960 comp rx=0 tx=0).stop 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.387+0000 7f489d0e8640 1 -- 172.21.15.6:0/1616854510 shutdown_connections 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.387+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1616854510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 0x7f48980af580 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.387+0000 7f489d0e8640 1 -- 172.21.15.6:0/1616854510 >> 172.21.15.6:0/1616854510 conn(0x7f489806b1c0 msgr2=0x7f489806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.387+0000 7f489d0e8640 1 -- 172.21.15.6:0/1616854510 shutdown_connections 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.387+0000 7f489d0e8640 1 -- 172.21.15.6:0/1616854510 wait complete. 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.388+0000 7f489d0e8640 1 Processor -- start 2024-03-18T14:42:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.388+0000 7f489d0e8640 1 -- start start 2024-03-18T14:42:24.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f489d0e8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4898071ad0 0x7f4898141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f489d0e8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 0x7f4898142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f489d0e8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489815dac0 0x7f48981e51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f489d0e8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4898159be0 con 0x7f4898071ad0 2024-03-18T14:42:24.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f489d0e8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4898159a60 con 0x7f489815cf10 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f489d0e8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f48980afac0 con 0x7f489815dac0 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4898071ad0 0x7f4898141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4898071ad0 0x7f4898141ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36784/0 (socket says 172.21.15.6:36784) 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 -- 172.21.15.6:0/1512315132 learned_addr learned my addr 172.21.15.6:0/1512315132 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f488ffff640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 0x7f4898142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f489759e640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489815dac0 0x7f48981e51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 -- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489815dac0 msgr2=0x7f48981e51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489815dac0 0x7f48981e51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 -- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 msgr2=0x7f4898142cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 0x7f4898142cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 -- 172.21.15.6:0/1512315132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4898140c40 con 0x7f4898071ad0 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.389+0000 7f4896d9d640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4898071ad0 0x7f4898141ce0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f488800c500 tx=0x7f488800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.390+0000 7f488f7fe640 1 -- 172.21.15.6:0/1512315132 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880052d0 con 0x7f4898071ad0 2024-03-18T14:42:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.390+0000 7f488f7fe640 1 -- 172.21.15.6:0/1512315132 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4888005490 con 0x7f4898071ad0 2024-03-18T14:42:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.390+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4898140f20 con 0x7f4898071ad0 2024-03-18T14:42:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.390+0000 7f488f7fe640 1 -- 172.21.15.6:0/1512315132 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888012870 con 0x7f4898071ad0 2024-03-18T14:42:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.390+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48981414e0 con 0x7f4898071ad0 2024-03-18T14:42:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.391+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48980751b0 con 0x7f4898071ad0 2024-03-18T14:42:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.391+0000 7f488f7fe640 1 -- 172.21.15.6:0/1512315132 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4888012a10 con 0x7f4898071ad0 2024-03-18T14:42:24.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.391+0000 7f488f7fe640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.391+0000 7f488f7fe640 1 -- 172.21.15.6:0/1512315132 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f488808aa10 con 0x7f4898071ad0 2024-03-18T14:42:24.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.392+0000 7f488ffff640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.392+0000 7f488ffff640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4880036000 tx=0x7f488003b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.396+0000 7f488f7fe640 1 -- 172.21.15.6:0/1512315132 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4888058ac0 con 0x7f4898071ad0 2024-03-18T14:42:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.580+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4898072390 con 0x7f4898071ad0 2024-03-18T14:42:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.580+0000 7f488f7fe640 1 -- 172.21.15.6:0/1512315132 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f488805cb60 con 0x7f4898071ad0 2024-03-18T14:42:24.583 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 msgr2=0x7f487806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4880036000 tx=0x7f488003b040 comp rx=0 tx=0).stop 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4898071ad0 msgr2=0x7f4898141ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4898071ad0 0x7f4898141ce0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f488800c500 tx=0x7f488800c9c0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 shutdown_connections 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489815dac0 0x7f48981e51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f489815cf10 0x7f4898142cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 --2- 172.21.15.6:0/1512315132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4898071ad0 0x7f4898141ce0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 >> 172.21.15.6:0/1512315132 conn(0x7f489806b1c0 msgr2=0x7f4898145310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 shutdown_connections 2024-03-18T14:42:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.582+0000 7f489d0e8640 1 -- 172.21.15.6:0/1512315132 wait complete. 2024-03-18T14:42:24.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.698+0000 7faed28d1640 1 Processor -- start 2024-03-18T14:42:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.698+0000 7faed28d1640 1 -- start start 2024-03-18T14:42:24.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faed28d1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc15be60 0x7faecc0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faed28d1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faecc0cf930 0x7faecc0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faed28d1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faecc06fcc0 con 0x7faecc0d0670 2024-03-18T14:42:24.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faed28d1640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7faecc06ffa0 con 0x7faecc0cf930 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faed28d1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faecc070280 con 0x7faecc15be60 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faecbfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc15be60 0x7faecc0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faecbfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc15be60 0x7faecc0ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50896/0 (socket says 172.21.15.6:50896) 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faecbfff640 1 -- 172.21.15.6:0/3562382544 learned_addr learned my addr 172.21.15.6:0/3562382544 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.699+0000 7faecb7fe640 1 --2- 172.21.15.6:0/3562382544 >> v2:172.21.15.92:3300/0 conn(0x7faecc0cf930 0x7faecc0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.700+0000 7faecbfff640 1 -- 172.21.15.6:0/3562382544 >> v2:172.21.15.92:3300/0 conn(0x7faecc0cf930 msgr2=0x7faecc0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.700+0000 7faecbfff640 1 --2- 172.21.15.6:0/3562382544 >> v2:172.21.15.92:3300/0 conn(0x7faecc0cf930 0x7faecc0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.700+0000 7faecbfff640 1 -- 172.21.15.6:0/3562382544 >> v1:172.21.15.92:6789/0 conn(0x7faecc0d0670 legacy=0x7faecc0cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.700+0000 7faecbfff640 1 -- 172.21.15.6:0/3562382544 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faecc070560 con 0x7faecc15be60 2024-03-18T14:42:24.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.700+0000 7faecb7fe640 1 --2- 172.21.15.6:0/3562382544 >> v2:172.21.15.92:3300/0 conn(0x7faecc0cf930 0x7faecc0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.700+0000 7faecbfff640 1 --2- 172.21.15.6:0/3562382544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc15be60 0x7faecc0ceb90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faebc009870 tx=0x7faebc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd19ec598d73813e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faecaffd640 1 -- 172.21.15.6:0/3562382544 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc035020 con 0x7faecc15be60 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faecaffd640 1 -- 172.21.15.6:0/3562382544 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faebc031640 con 0x7faecc15be60 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faecaffd640 1 -- 172.21.15.6:0/3562382544 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc02d040 con 0x7faecc15be60 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 -- 172.21.15.6:0/3562382544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc15be60 msgr2=0x7faecc0ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 --2- 172.21.15.6:0/3562382544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc15be60 0x7faecc0ceb90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faebc009870 tx=0x7faebc02b8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 -- 172.21.15.6:0/3562382544 shutdown_connections 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 --2- 172.21.15.6:0/3562382544 >> v2:172.21.15.92:3300/0 conn(0x7faecc0cf930 0x7faecc0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 --2- 172.21.15.6:0/3562382544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc15be60 0x7faecc0ceb90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 -- 172.21.15.6:0/3562382544 >> 172.21.15.6:0/3562382544 conn(0x7faecc0d1b50 msgr2=0x7faecc0d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 -- 172.21.15.6:0/3562382544 shutdown_connections 2024-03-18T14:42:24.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.701+0000 7faed28d1640 1 -- 172.21.15.6:0/3562382544 wait complete. 2024-03-18T14:42:24.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.702+0000 7faed28d1640 1 Processor -- start 2024-03-18T14:42:24.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed28d1640 1 -- start start 2024-03-18T14:42:24.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed28d1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc0cf930 0x7faecc0b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed28d1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0d0670 0x7faecc15e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed28d1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faecc15be60 0x7faecc152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed28d1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faecc0ae710 con 0x7faecc15be60 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed28d1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faecc15c240 con 0x7faecc0d0670 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed28d1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faecc0ae890 con 0x7faecc0cf930 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed0e6e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faecc15be60 0x7faecc152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed0e6e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faecc15be60 0x7faecc152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36792/0 (socket says 172.21.15.6:36792) 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faed0e6e640 1 -- 172.21.15.6:0/1447975578 learned_addr learned my addr 172.21.15.6:0/1447975578 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faecb7fe640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0d0670 0x7faecc15e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.703+0000 7faecbfff640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc0cf930 0x7faecc0b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed0e6e640 1 -- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc0cf930 msgr2=0x7faecc0b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed0e6e640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc0cf930 0x7faecc0b73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed0e6e640 1 -- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0d0670 msgr2=0x7faecc15e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed0e6e640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0d0670 0x7faecc15e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed0e6e640 1 -- 172.21.15.6:0/1447975578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faecc15ee00 con 0x7faecc15be60 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed0e6e640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faecc15be60 0x7faecc152ed0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faec400c6b0 tx=0x7faec400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faec8ff9640 1 -- 172.21.15.6:0/1447975578 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4003b70 con 0x7faecc15be60 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faecc15f0e0 con 0x7faecc15be60 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faec8ff9640 1 -- 172.21.15.6:0/1447975578 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec4003d30 con 0x7faecc15be60 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faec8ff9640 1 -- 172.21.15.6:0/1447975578 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4005310 con 0x7faecc15be60 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.704+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faecc0b4940 con 0x7faecc15be60 2024-03-18T14:42:24.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.705+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faecc0751b0 con 0x7faecc15be60 2024-03-18T14:42:24.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.706+0000 7faec8ff9640 1 -- 172.21.15.6:0/1447975578 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faec40054b0 con 0x7faecc15be60 2024-03-18T14:42:24.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.706+0000 7faec8ff9640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faeac06bbd0 0x7faeac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:24.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.706+0000 7faec8ff9640 1 -- 172.21.15.6:0/1447975578 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7faec408ab70 con 0x7faecc15be60 2024-03-18T14:42:24.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.706+0000 7faecbfff640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faeac06bbd0 0x7faeac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:24.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.706+0000 7faecbfff640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faeac06bbd0 0x7faeac06e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faebc036000 tx=0x7faebc03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:24.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.710+0000 7faec8ff9640 1 -- 172.21.15.6:0/1447975578 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faec4058c10 con 0x7faecc15be60 2024-03-18T14:42:24.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.891+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faecc0b4b10 con 0x7faecc15be60 2024-03-18T14:42:24.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.891+0000 7faec8ff9640 1 -- 172.21.15.6:0/1447975578 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7faec405ccb0 con 0x7faecc15be60 2024-03-18T14:42:24.894 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faeac06bbd0 msgr2=0x7faeac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faeac06bbd0 0x7faeac06e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faebc036000 tx=0x7faebc03c040 comp rx=0 tx=0).stop 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faecc15be60 msgr2=0x7faecc152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faecc15be60 0x7faecc152ed0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faec400c6b0 tx=0x7faec400cb70 comp rx=0 tx=0).stop 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 shutdown_connections 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faeac06bbd0 0x7faeac06e090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faecc15be60 0x7faecc152ed0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faecc0d0670 0x7faecc15e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 --2- 172.21.15.6:0/1447975578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faecc0cf930 0x7faecc0b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 >> 172.21.15.6:0/1447975578 conn(0x7faecc0d1b50 msgr2=0x7faecc0b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 shutdown_connections 2024-03-18T14:42:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:24.893+0000 7faed28d1640 1 -- 172.21.15.6:0/1447975578 wait complete. 2024-03-18T14:42:24.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-18T14:42:25.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.006+0000 7f9e12176640 1 Processor -- start 2024-03-18T14:42:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.006+0000 7f9e12176640 1 -- start start 2024-03-18T14:42:25.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.006+0000 7f9e12176640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0b17b0 0x7f9e0c0b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.006+0000 7f9e12176640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9e0c0d0260 0x7f9e0c0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.006+0000 7f9e12176640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e0c0cfd80 con 0x7f9e0c06cce0 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.006+0000 7f9e12176640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9e0c0b2df0 con 0x7f9e0c0d0260 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.006+0000 7f9e12176640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0c0b2270 con 0x7f9e0c0b17b0 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9e0c06cce0 0x7f9e0c06d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55908/0 (socket says 172.21.15.6:55908) 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0b7fe640 1 -- 172.21.15.6:0/993603335 learned_addr learned my addr 172.21.15.6:0/993603335 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0affd640 1 --2- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0b17b0 0x7f9e0c0b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0bfff640 1 --2- 172.21.15.6:0/993603335 >> v2:172.21.15.92:3300/0 conn(0x7f9e0c0d0260 0x7f9e0c0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0a7fc640 1 -- 172.21.15.6:0/993603335 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1767699299 0 0) 0x7f9e0c0cfd80 con 0x7f9e0c06cce0 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0a7fc640 1 -- 172.21.15.6:0/993603335 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df4003520 con 0x7f9e0c06cce0 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0bfff640 1 -- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0b17b0 msgr2=0x7f9e0c0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0bfff640 1 --2- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0b17b0 0x7f9e0c0b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0bfff640 1 -- 172.21.15.6:0/993603335 >> v1:172.21.15.6:6789/0 conn(0x7f9e0c06cce0 legacy=0x7f9e0c06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0bfff640 1 -- 172.21.15.6:0/993603335 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0c0b2550 con 0x7f9e0c0d0260 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.007+0000 7f9e0affd640 1 --2- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0b17b0 0x7f9e0c0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:25.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e0bfff640 1 --2- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0d0260 0x7f9e0c0d06f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009d90 tx=0x7f9e0402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7453b14873eda89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:25.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e0a7fc640 1 -- 172.21.15.6:0/993603335 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04039070 con 0x7f9e0c0d0260 2024-03-18T14:42:25.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e0a7fc640 1 -- 172.21.15.6:0/993603335 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0402cbc0 con 0x7f9e0c0d0260 2024-03-18T14:42:25.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e0a7fc640 1 -- 172.21.15.6:0/993603335 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04034440 con 0x7f9e0c0d0260 2024-03-18T14:42:25.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e12176640 1 -- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0d0260 msgr2=0x7f9e0c0d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e12176640 1 --2- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0d0260 0x7f9e0c0d06f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009d90 tx=0x7f9e0402be10 comp rx=0 tx=0).stop 2024-03-18T14:42:25.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e12176640 1 -- 172.21.15.6:0/993603335 shutdown_connections 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e12176640 1 --2- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0d0260 0x7f9e0c0d06f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e12176640 1 --2- 172.21.15.6:0/993603335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0b17b0 0x7f9e0c0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.008+0000 7f9e12176640 1 -- 172.21.15.6:0/993603335 >> 172.21.15.6:0/993603335 conn(0x7f9e0c08b130 msgr2=0x7f9e0c08b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.009+0000 7f9e12176640 1 -- 172.21.15.6:0/993603335 shutdown_connections 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.009+0000 7f9e12176640 1 -- 172.21.15.6:0/993603335 wait complete. 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 Processor -- start 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 -- start start 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c06cce0 0x7f9e0c158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0b17b0 0x7f9e0c15fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0d0260 0x7f9e0c15e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c15e330 con 0x7f9e0c06cce0 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c0b2b10 con 0x7f9e0c0b17b0 2024-03-18T14:42:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.010+0000 7f9e12176640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0c06c690 con 0x7f9e0c0d0260 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c06cce0 0x7f9e0c158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c06cce0 0x7f9e0c158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36804/0 (socket says 172.21.15.6:36804) 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 -- 172.21.15.6:0/1167792453 learned_addr learned my addr 172.21.15.6:0/1167792453 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0affd640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0b17b0 0x7f9e0c15fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0bfff640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0d0260 0x7f9e0c15e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 -- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0d0260 msgr2=0x7f9e0c15e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0d0260 0x7f9e0c15e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 -- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0b17b0 msgr2=0x7f9e0c15fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0b17b0 0x7f9e0c15fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 -- 172.21.15.6:0/1167792453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0c1605a0 con 0x7f9e0c06cce0 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e0b7fe640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c06cce0 0x7f9e0c158e90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00c430 tx=0x7f9dfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9df3fff640 1 -- 172.21.15.6:0/1167792453 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc017070 con 0x7f9e0c06cce0 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9df3fff640 1 -- 172.21.15.6:0/1167792453 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc005ce0 con 0x7f9e0c06cce0 2024-03-18T14:42:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9df3fff640 1 -- 172.21.15.6:0/1167792453 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0072f0 con 0x7f9e0c06cce0 2024-03-18T14:42:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0c160880 con 0x7f9e0c06cce0 2024-03-18T14:42:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.011+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0c1593d0 con 0x7f9e0c06cce0 2024-03-18T14:42:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.012+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0c09fa80 con 0x7f9e0c06cce0 2024-03-18T14:42:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.013+0000 7f9df3fff640 1 -- 172.21.15.6:0/1167792453 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9dfc01a070 con 0x7f9e0c06cce0 2024-03-18T14:42:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.013+0000 7f9df3fff640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9df806bbd0 0x7f9df806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.013+0000 7f9df3fff640 1 -- 172.21.15.6:0/1167792453 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9dfc08a280 con 0x7f9e0c06cce0 2024-03-18T14:42:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.013+0000 7f9e0affd640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9df806bbd0 0x7f9df806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.014+0000 7f9e0affd640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9df806bbd0 0x7f9df806e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009560 tx=0x7f9e00009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:25.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.017+0000 7f9df3fff640 1 -- 172.21.15.6:0/1167792453 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9dfc058320 con 0x7f9e0c06cce0 2024-03-18T14:42:25.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.270+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7f9e0c0b7800 con 0x7f9e0c06cce0 2024-03-18T14:42:25.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.693+0000 7f9df3fff640 1 -- 172.21.15.6:0/1167792453 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v72) ==== 150+0+0 (secure 0 0 0) 0x7f9dfc05c3c0 con 0x7f9e0c06cce0 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9df806bbd0 msgr2=0x7f9df806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9df806bbd0 0x7f9df806e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009560 tx=0x7f9e00009190 comp rx=0 tx=0).stop 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c06cce0 msgr2=0x7f9e0c158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c06cce0 0x7f9e0c158e90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00c430 tx=0x7f9dfc00c8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 shutdown_connections 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9df806bbd0 0x7f9df806e090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e0c0d0260 0x7f9e0c15e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c0b17b0 0x7f9e0c15fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 --2- 172.21.15.6:0/1167792453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c06cce0 0x7f9e0c158e90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 >> 172.21.15.6:0/1167792453 conn(0x7f9e0c08b130 msgr2=0x7f9e0c0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:25.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 shutdown_connections 2024-03-18T14:42:25.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.695+0000 7f9e12176640 1 -- 172.21.15.6:0/1167792453 wait complete. 2024-03-18T14:42:25.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T14:42:25.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.813+0000 7fd07f997640 1 Processor -- start 2024-03-18T14:42:25.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.814+0000 7fd07f997640 1 -- start start 2024-03-18T14:42:25.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.814+0000 7fd07f997640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0780b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.814+0000 7fd07f997640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0780ae6f0 0x7fd0780d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.814+0000 7fd07f997640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0780aeb50 con 0x7fd0780d0650 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.814+0000 7fd07f997640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd0780aedf0 con 0x7fd0780ae6f0 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.814+0000 7fd07f997640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd0780af0d0 con 0x7fd07815bdb0 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07cf32640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0780ae6f0 0x7fd0780d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07df34640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd0780d0650 0x7fd0780b1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55910/0 (socket says 172.21.15.6:55910) 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07df34640 1 -- 172.21.15.6:0/1195781211 learned_addr learned my addr 172.21.15.6:0/1195781211 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07d733640 1 --2- 172.21.15.6:0/1195781211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0780b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07cf32640 1 -- 172.21.15.6:0/1195781211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 msgr2=0x7fd0780b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07cf32640 1 --2- 172.21.15.6:0/1195781211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0780b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07cf32640 1 -- 172.21.15.6:0/1195781211 >> v1:172.21.15.6:6789/0 conn(0x7fd0780d0650 legacy=0x7fd0780b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd07cf32640 1 -- 172.21.15.6:0/1195781211 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0780af3b0 con 0x7fd0780ae6f0 2024-03-18T14:42:25.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.815+0000 7fd06ffff640 1 -- 172.21.15.6:0/1195781211 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826074883 0 0) 0x7fd0780aeb50 con 0x7fd0780d0650 2024-03-18T14:42:25.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07cf32640 1 --2- 172.21.15.6:0/1195781211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd0780d0110 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd0680097a0 tx=0x7fd06802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0433a49da7d1f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd06ffff640 1 -- 172.21.15.6:0/1195781211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd068035020 con 0x7fd0780ae6f0 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd06ffff640 1 -- 172.21.15.6:0/1195781211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd068031830 con 0x7fd0780ae6f0 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd06ffff640 1 -- 172.21.15.6:0/1195781211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd068031c00 con 0x7fd0780ae6f0 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 -- 172.21.15.6:0/1195781211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 msgr2=0x7fd0780d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 --2- 172.21.15.6:0/1195781211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd0780d0110 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd0680097a0 tx=0x7fd06802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 -- 172.21.15.6:0/1195781211 shutdown_connections 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 --2- 172.21.15.6:0/1195781211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd0780d0110 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 --2- 172.21.15.6:0/1195781211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0780b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 -- 172.21.15.6:0/1195781211 >> 172.21.15.6:0/1195781211 conn(0x7fd07809ae00 msgr2=0x7fd07808b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 -- 172.21.15.6:0/1195781211 shutdown_connections 2024-03-18T14:42:25.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.816+0000 7fd07f997640 1 -- 172.21.15.6:0/1195781211 wait complete. 2024-03-18T14:42:25.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 Processor -- start 2024-03-18T14:42:25.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 -- start start 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd078150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780d0650 0x7fd078151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0781f51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd07815c310 con 0x7fd0780ae6f0 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd07815c190 con 0x7fd0780d0650 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07f997640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd0780b0ea0 con 0x7fd07815bdb0 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07d733640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd078150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07d733640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd078150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36814/0 (socket says 172.21.15.6:36814) 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07d733640 1 -- 172.21.15.6:0/3653323600 learned_addr learned my addr 172.21.15.6:0/3653323600 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.818+0000 7fd07cf32640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780d0650 0x7fd078151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07df34640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0781f51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07d733640 1 -- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 msgr2=0x7fd0781f51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07d733640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0781f51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07d733640 1 -- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780d0650 msgr2=0x7fd078151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07d733640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780d0650 0x7fd078151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07d733640 1 -- 172.21.15.6:0/3653323600 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0780b5f90 con 0x7fd0780ae6f0 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07d733640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd078150d50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e870 tx=0x7fd07400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd06dffb640 1 -- 172.21.15.6:0/3653323600 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd074017070 con 0x7fd0780ae6f0 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd06dffb640 1 -- 172.21.15.6:0/3653323600 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd07400c940 con 0x7fd0780ae6f0 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd06dffb640 1 -- 172.21.15.6:0/3653323600 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07400cd30 con 0x7fd0780ae6f0 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0780ca2c0 con 0x7fd0780ae6f0 2024-03-18T14:42:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.819+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0780b6210 con 0x7fd0780ae6f0 2024-03-18T14:42:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.820+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd07804f5b0 con 0x7fd0780ae6f0 2024-03-18T14:42:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.821+0000 7fd06dffb640 1 -- 172.21.15.6:0/3653323600 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd07401a070 con 0x7fd0780ae6f0 2024-03-18T14:42:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.821+0000 7fd06dffb640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd05c06bbd0 0x7fd05c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.821+0000 7fd06dffb640 1 -- 172.21.15.6:0/3653323600 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd07408ac90 con 0x7fd0780ae6f0 2024-03-18T14:42:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.821+0000 7fd07cf32640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd05c06bbd0 0x7fd05c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.822+0000 7fd07cf32640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd05c06bbd0 0x7fd05c06e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd068036000 tx=0x7fd068008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:25.825+0000 7fd06dffb640 1 -- 172.21.15.6:0/3653323600 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd0780b5f90 con 0x7fd0780ae6f0 2024-03-18T14:42:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.011+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd07808b180 con 0x7fd0780ae6f0 2024-03-18T14:42:26.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.014+0000 7fd06dffb640 1 -- 172.21.15.6:0/3653323600 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fd074058d30 con 0x7fd0780ae6f0 2024-03-18T14:42:26.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd05c06bbd0 msgr2=0x7fd05c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd05c06bbd0 0x7fd05c06e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd068036000 tx=0x7fd068008030 comp rx=0 tx=0).stop 2024-03-18T14:42:26.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 msgr2=0x7fd078150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd078150d50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e870 tx=0x7fd07400ed30 comp rx=0 tx=0).stop 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 shutdown_connections 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd05c06bbd0 0x7fd05c06e090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd07815bdb0 0x7fd0781f51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0780d0650 0x7fd078151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 --2- 172.21.15.6:0/3653323600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0780ae6f0 0x7fd078150d50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 >> 172.21.15.6:0/3653323600 conn(0x7fd07809ae00 msgr2=0x7fd0780b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 shutdown_connections 2024-03-18T14:42:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.016+0000 7fd07f997640 1 -- 172.21.15.6:0/3653323600 wait complete. 2024-03-18T14:42:26.030 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T14:42:26.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:42:26.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:42:26.121 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T14:42:26.121 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:42:26.121 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:26.136 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:42:26.136 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:26.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:26.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.332+0000 7fe8b746e640 1 Processor -- start 2024-03-18T14:42:26.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.332+0000 7fe8b746e640 1 -- start start 2024-03-18T14:42:26.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.333+0000 7fe8b746e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe8b015e020 0x7fe8b0162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.333+0000 7fe8b746e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b00b9660 con 0x7fe8b00ced00 2024-03-18T14:42:26.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.333+0000 7fe8b746e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b00ce390 con 0x7fe8b00d3210 2024-03-18T14:42:26.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.333+0000 7fe8b746e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe8b00d2270 con 0x7fe8b015e020 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.333+0000 7fe8b4a09640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe8b00ced00 0x7fe8b00d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55926/0 (socket says 172.21.15.6:55926) 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.333+0000 7fe8b4a09640 1 -- 172.21.15.6:0/444511375 learned_addr learned my addr 172.21.15.6:0/444511375 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.333+0000 7fe8b520a640 1 --2- 172.21.15.6:0/444511375 >> v2:172.21.15.92:3300/0 conn(0x7fe8b015e020 0x7fe8b0162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.334+0000 7fe8a3fff640 1 -- 172.21.15.6:0/444511375 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456043229 0 0) 0x7fe8b00b9660 con 0x7fe8b00ced00 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.334+0000 7fe8a3fff640 1 -- 172.21.15.6:0/444511375 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe89c003520 con 0x7fe8b00ced00 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.334+0000 7fe8a3fff640 1 -- 172.21.15.6:0/444511375 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268430453 0 0) 0x7fe8b00ce390 con 0x7fe8b00d3210 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.334+0000 7fe8a3fff640 1 -- 172.21.15.6:0/444511375 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8b00b9660 con 0x7fe8b00d3210 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.334+0000 7fe8b520a640 1 -- 172.21.15.6:0/444511375 >> v1:172.21.15.92:6789/0 conn(0x7fe8b00d3210 legacy=0x7fe8b00d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.334+0000 7fe8b520a640 1 -- 172.21.15.6:0/444511375 >> v1:172.21.15.6:6789/0 conn(0x7fe8b00ced00 legacy=0x7fe8b00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.334+0000 7fe8b520a640 1 -- 172.21.15.6:0/444511375 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b00b9900 con 0x7fe8b015e020 2024-03-18T14:42:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8b520a640 1 --2- 172.21.15.6:0/444511375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b015e020 0x7fe8b0162380 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0097a0 tx=0x7fe8ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ca1eface4759080 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8a3fff640 1 -- 172.21.15.6:0/444511375 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac035020 con 0x7fe8b015e020 2024-03-18T14:42:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8a3fff640 1 -- 172.21.15.6:0/444511375 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac031830 con 0x7fe8b015e020 2024-03-18T14:42:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8a3fff640 1 -- 172.21.15.6:0/444511375 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac031be0 con 0x7fe8b015e020 2024-03-18T14:42:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8b746e640 1 -- 172.21.15.6:0/444511375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b015e020 msgr2=0x7fe8b0162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8b746e640 1 --2- 172.21.15.6:0/444511375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b015e020 0x7fe8b0162380 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0097a0 tx=0x7fe8ac02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8b746e640 1 -- 172.21.15.6:0/444511375 shutdown_connections 2024-03-18T14:42:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8b746e640 1 --2- 172.21.15.6:0/444511375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b015e020 0x7fe8b0162380 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.335+0000 7fe8b746e640 1 -- 172.21.15.6:0/444511375 >> 172.21.15.6:0/444511375 conn(0x7fe8b0072ce0 msgr2=0x7fe8b00730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.336+0000 7fe8b746e640 1 -- 172.21.15.6:0/444511375 shutdown_connections 2024-03-18T14:42:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.336+0000 7fe8b746e640 1 -- 172.21.15.6:0/444511375 wait complete. 2024-03-18T14:42:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 Processor -- start 2024-03-18T14:42:26.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 -- start start 2024-03-18T14:42:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8b00ced00 0x7fe8b015ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b00d3210 0x7fe8b015ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe8b015e020 0x7fe8b01604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b00cf2c0 con 0x7fe8b00ced00 2024-03-18T14:42:26.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b00ce890 con 0x7fe8b00d3210 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.337+0000 7fe8b746e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe8b00cf440 con 0x7fe8b015e020 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8b00ced00 0x7fe8b015ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8b00ced00 0x7fe8b015ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36828/0 (socket says 172.21.15.6:36828) 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 -- 172.21.15.6:0/2512940315 learned_addr learned my addr 172.21.15.6:0/2512940315 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b5a0b640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe8b015e020 0x7fe8b01604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b4a09640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b00d3210 0x7fe8b015ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 -- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe8b015e020 msgr2=0x7fe8b01604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe8b015e020 0x7fe8b01604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 -- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b00d3210 msgr2=0x7fe8b015ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b00d3210 0x7fe8b015ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 -- 172.21.15.6:0/2512940315 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b015f520 con 0x7fe8b00ced00 2024-03-18T14:42:26.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b520a640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8b00ced00 0x7fe8b015ecd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000c00 tx=0x7fe8ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8a1ffb640 1 -- 172.21.15.6:0/2512940315 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac040070 con 0x7fe8b00ced00 2024-03-18T14:42:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8a1ffb640 1 -- 172.21.15.6:0/2512940315 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac03e070 con 0x7fe8b00ced00 2024-03-18T14:42:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b015f800 con 0x7fe8b00ced00 2024-03-18T14:42:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.338+0000 7fe8a1ffb640 1 -- 172.21.15.6:0/2512940315 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac035040 con 0x7fe8b00ced00 2024-03-18T14:42:26.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.339+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b0153310 con 0x7fe8b00ced00 2024-03-18T14:42:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.339+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b008afe0 con 0x7fe8b00ced00 2024-03-18T14:42:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.340+0000 7fe8a1ffb640 1 -- 172.21.15.6:0/2512940315 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe8ac038020 con 0x7fe8b00ced00 2024-03-18T14:42:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.340+0000 7fe8a1ffb640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe89406bbd0 0x7fe89406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.340+0000 7fe8a1ffb640 1 -- 172.21.15.6:0/2512940315 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe8ac0ac090 con 0x7fe8b00ced00 2024-03-18T14:42:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.341+0000 7fe8b4a09640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe89406bbd0 0x7fe89406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.341+0000 7fe8b4a09640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe89406bbd0 0x7fe89406e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe8b00cad90 tx=0x7fe8a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.345+0000 7fe8a1ffb640 1 -- 172.21.15.6:0/2512940315 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe8ac07a1c0 con 0x7fe8b00ced00 2024-03-18T14:42:26.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.529+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8b0156520 con 0x7fe8b00ced00 2024-03-18T14:42:26.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.530+0000 7fe8a1ffb640 1 -- 172.21.15.6:0/2512940315 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fe8ac07e260 con 0x7fe8b00ced00 2024-03-18T14:42:26.533 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:26.533 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:26.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe89406bbd0 msgr2=0x7fe89406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe89406bbd0 0x7fe89406e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe8b00cad90 tx=0x7fe8a4009040 comp rx=0 tx=0).stop 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8b00ced00 msgr2=0x7fe8b015ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8b00ced00 0x7fe8b015ecd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000c00 tx=0x7fe8ac0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 shutdown_connections 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe89406bbd0 0x7fe89406e090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe8b015e020 0x7fe8b01604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8b00d3210 0x7fe8b015ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 --2- 172.21.15.6:0/2512940315 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8b00ced00 0x7fe8b015ecd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 >> 172.21.15.6:0/2512940315 conn(0x7fe8b0072ce0 msgr2=0x7fe8b00d3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 shutdown_connections 2024-03-18T14:42:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.533+0000 7fe8b746e640 1 -- 172.21.15.6:0/2512940315 wait complete. 2024-03-18T14:42:26.546 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:26.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.649+0000 7fc2ca780640 1 Processor -- start 2024-03-18T14:42:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.649+0000 7fc2ca780640 1 -- start start 2024-03-18T14:42:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.650+0000 7fc2ca780640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2c4072a70 0x7fc2c415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.650+0000 7fc2ca780640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2c40cee70 con 0x7fc2c4154850 2024-03-18T14:42:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.650+0000 7fc2ca780640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2c40cf150 con 0x7fc2c415bae0 2024-03-18T14:42:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.650+0000 7fc2ca780640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc2c40cf430 con 0x7fc2c4072a70 2024-03-18T14:42:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.650+0000 7fc2c3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc2c4154850 0x7fc2c40ce5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55934/0 (socket says 172.21.15.6:55934) 2024-03-18T14:42:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.650+0000 7fc2c3fff640 1 -- 172.21.15.6:0/3636116249 learned_addr learned my addr 172.21.15.6:0/3636116249 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.650+0000 7fc2c8d1d640 1 --2- 172.21.15.6:0/3636116249 >> v2:172.21.15.92:3300/0 conn(0x7fc2c4072a70 0x7fc2c415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.651+0000 7fc2c2ffd640 1 -- 172.21.15.6:0/3636116249 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972466243 0 0) 0x7fc2c40cee70 con 0x7fc2c4154850 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.651+0000 7fc2c2ffd640 1 -- 172.21.15.6:0/3636116249 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2ac003440 con 0x7fc2c4154850 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.651+0000 7fc2c2ffd640 1 -- 172.21.15.6:0/3636116249 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3655985397 0 0) 0x7fc2c40cf150 con 0x7fc2c415bae0 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.651+0000 7fc2c8d1d640 1 -- 172.21.15.6:0/3636116249 >> v1:172.21.15.92:6789/0 conn(0x7fc2c415bae0 legacy=0x7fc2c4072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.651+0000 7fc2c8d1d640 1 -- 172.21.15.6:0/3636116249 >> v1:172.21.15.6:6789/0 conn(0x7fc2c4154850 legacy=0x7fc2c40ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.651+0000 7fc2c8d1d640 1 -- 172.21.15.6:0/3636116249 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c40cbd80 con 0x7fc2c4072a70 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2c8d1d640 1 --2- 172.21.15.6:0/3636116249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4072a70 0x7fc2c415b4c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009d90 tx=0x7fc2bc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=292182a8098c6a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2c2ffd640 1 -- 172.21.15.6:0/3636116249 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc039070 con 0x7fc2c4072a70 2024-03-18T14:42:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2c2ffd640 1 -- 172.21.15.6:0/3636116249 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2bc02cd00 con 0x7fc2c4072a70 2024-03-18T14:42:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2c2ffd640 1 -- 172.21.15.6:0/3636116249 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc034660 con 0x7fc2c4072a70 2024-03-18T14:42:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2ca780640 1 -- 172.21.15.6:0/3636116249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4072a70 msgr2=0x7fc2c415b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2ca780640 1 --2- 172.21.15.6:0/3636116249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4072a70 0x7fc2c415b4c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009d90 tx=0x7fc2bc02bf50 comp rx=0 tx=0).stop 2024-03-18T14:42:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2ca780640 1 -- 172.21.15.6:0/3636116249 shutdown_connections 2024-03-18T14:42:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2ca780640 1 --2- 172.21.15.6:0/3636116249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4072a70 0x7fc2c415b4c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2ca780640 1 -- 172.21.15.6:0/3636116249 >> 172.21.15.6:0/3636116249 conn(0x7fc2c406b1c0 msgr2=0x7fc2c406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2ca780640 1 -- 172.21.15.6:0/3636116249 shutdown_connections 2024-03-18T14:42:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.652+0000 7fc2ca780640 1 -- 172.21.15.6:0/3636116249 wait complete. 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.653+0000 7fc2ca780640 1 Processor -- start 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.654+0000 7fc2ca780640 1 -- start start 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.654+0000 7fc2ca780640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2c4072a70 0x7fc2c4149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.654+0000 7fc2ca780640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4154850 0x7fc2c41497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.654+0000 7fc2ca780640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c415bae0 0x7fc2c41f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.654+0000 7fc2ca780640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c4071c50 con 0x7fc2c4072a70 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.654+0000 7fc2ca780640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c4071ad0 con 0x7fc2c4154850 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.654+0000 7fc2ca780640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c4071dd0 con 0x7fc2c415bae0 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2c4072a70 0x7fc2c4149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2c4072a70 0x7fc2c4149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36838/0 (socket says 172.21.15.6:36838) 2024-03-18T14:42:26.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 -- 172.21.15.6:0/343293631 learned_addr learned my addr 172.21.15.6:0/343293631 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c37fe640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4154850 0x7fc2c41497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c8d1d640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c415bae0 0x7fc2c41f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 -- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c415bae0 msgr2=0x7fc2c41f72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c415bae0 0x7fc2c41f72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 -- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4154850 msgr2=0x7fc2c41497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4154850 0x7fc2c41497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 -- 172.21.15.6:0/343293631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c40caa20 con 0x7fc2c4072a70 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c3fff640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2c4072a70 0x7fc2c4149290 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800c460 tx=0x7fc2b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c0ff9640 1 -- 172.21.15.6:0/343293631 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8017070 con 0x7fc2c4072a70 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c0ff9640 1 -- 172.21.15.6:0/343293631 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2b8005ce0 con 0x7fc2c4072a70 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2c0ff9640 1 -- 172.21.15.6:0/343293631 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8007290 con 0x7fc2c4072a70 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c40cad00 con 0x7fc2c4072a70 2024-03-18T14:42:26.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.655+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c40cb2c0 con 0x7fc2c4072a70 2024-03-18T14:42:26.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.656+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2c40751b0 con 0x7fc2c4072a70 2024-03-18T14:42:26.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.657+0000 7fc2c0ff9640 1 -- 172.21.15.6:0/343293631 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc2b801a070 con 0x7fc2c4072a70 2024-03-18T14:42:26.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.657+0000 7fc2c0ff9640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2b006bbd0 0x7fc2b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.657+0000 7fc2c0ff9640 1 -- 172.21.15.6:0/343293631 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc2b808a1f0 con 0x7fc2c4072a70 2024-03-18T14:42:26.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.657+0000 7fc2c37fe640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2b006bbd0 0x7fc2b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.658+0000 7fc2c37fe640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2b006bbd0 0x7fc2b006e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2c41f8130 tx=0x7fc2b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.661+0000 7fc2c0ff9640 1 -- 172.21.15.6:0/343293631 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc2b8058290 con 0x7fc2c4072a70 2024-03-18T14:42:26.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.843+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc2c415cdb0 con 0x7fc2c4072a70 2024-03-18T14:42:26.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.844+0000 7fc2c0ff9640 1 -- 172.21.15.6:0/343293631 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fc2b805c330 con 0x7fc2c4072a70 2024-03-18T14:42:26.847 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2b006bbd0 msgr2=0x7fc2b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2b006bbd0 0x7fc2b006e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc2c41f8130 tx=0x7fc2b4009040 comp rx=0 tx=0).stop 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2c4072a70 msgr2=0x7fc2c4149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2c4072a70 0x7fc2c4149290 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800c460 tx=0x7fc2b800c920 comp rx=0 tx=0).stop 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 shutdown_connections 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2b006bbd0 0x7fc2b006e090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c415bae0 0x7fc2c41f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c4154850 0x7fc2c41497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 --2- 172.21.15.6:0/343293631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2c4072a70 0x7fc2c4149290 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 >> 172.21.15.6:0/343293631 conn(0x7fc2c406b1c0 msgr2=0x7fc2c4151b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:26.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 shutdown_connections 2024-03-18T14:42:26.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.846+0000 7fc2ca780640 1 -- 172.21.15.6:0/343293631 wait complete. 2024-03-18T14:42:26.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:26.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.962+0000 7fa946473640 1 Processor -- start 2024-03-18T14:42:26.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.962+0000 7fa946473640 1 -- start start 2024-03-18T14:42:26.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa946473640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa94006b810 0x7fa9400ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa946473640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9400b22c0 con 0x7fa94006b430 2024-03-18T14:42:26.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa946473640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94015e650 con 0x7fa94015d640 2024-03-18T14:42:26.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa946473640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa94015e930 con 0x7fa94006b810 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa93ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa94006b430 0x7fa9400b1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55948/0 (socket says 172.21.15.6:55948) 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa93ffff640 1 -- 172.21.15.6:0/1446716852 learned_addr learned my addr 172.21.15.6:0/1446716852 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa93f7fe640 1 --2- 172.21.15.6:0/1446716852 >> v2:172.21.15.92:3300/0 conn(0x7fa94006b810 0x7fa9400ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.963+0000 7fa93effd640 1 -- 172.21.15.6:0/1446716852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1952466366 0 0) 0x7fa9400b22c0 con 0x7fa94006b430 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.964+0000 7fa93effd640 1 -- 172.21.15.6:0/1446716852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa928003520 con 0x7fa94006b430 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.964+0000 7fa93effd640 1 -- 172.21.15.6:0/1446716852 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142533122 0 0) 0x7fa94015e650 con 0x7fa94015d640 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.964+0000 7fa93effd640 1 -- 172.21.15.6:0/1446716852 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9400b22c0 con 0x7fa94015d640 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.964+0000 7fa93f7fe640 1 -- 172.21.15.6:0/1446716852 >> v1:172.21.15.92:6789/0 conn(0x7fa94015d640 legacy=0x7fa94015daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.964+0000 7fa93f7fe640 1 -- 172.21.15.6:0/1446716852 >> v1:172.21.15.6:6789/0 conn(0x7fa94006b430 legacy=0x7fa9400b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.964+0000 7fa93f7fe640 1 -- 172.21.15.6:0/1446716852 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94015ec10 con 0x7fa94006b810 2024-03-18T14:42:26.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.964+0000 7fa93f7fe640 1 --2- 172.21.15.6:0/1446716852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9400ce670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa9300097a0 tx=0x7fa93002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59d3482694fff15d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa93effd640 1 -- 172.21.15.6:0/1446716852 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930034070 con 0x7fa94006b810 2024-03-18T14:42:26.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa93effd640 1 -- 172.21.15.6:0/1446716852 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa930030610 con 0x7fa94006b810 2024-03-18T14:42:26.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa93effd640 1 -- 172.21.15.6:0/1446716852 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9300309e0 con 0x7fa94006b810 2024-03-18T14:42:26.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa946473640 1 -- 172.21.15.6:0/1446716852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 msgr2=0x7fa9400ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa946473640 1 --2- 172.21.15.6:0/1446716852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9400ce670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa9300097a0 tx=0x7fa93002b960 comp rx=0 tx=0).stop 2024-03-18T14:42:26.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa946473640 1 -- 172.21.15.6:0/1446716852 shutdown_connections 2024-03-18T14:42:26.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa946473640 1 --2- 172.21.15.6:0/1446716852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9400ce670 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa946473640 1 -- 172.21.15.6:0/1446716852 >> 172.21.15.6:0/1446716852 conn(0x7fa940099a50 msgr2=0x7fa9400d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:26.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa946473640 1 -- 172.21.15.6:0/1446716852 shutdown_connections 2024-03-18T14:42:26.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.965+0000 7fa946473640 1 -- 172.21.15.6:0/1446716852 wait complete. 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.966+0000 7fa946473640 1 Processor -- start 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa946473640 1 -- start start 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa946473640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa94006b430 0x7fa9400ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa946473640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9401606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa946473640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa94015d640 0x7fa940160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa946473640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa9400cce70 con 0x7fa94015d640 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa946473640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa9400cccf0 con 0x7fa94006b810 2024-03-18T14:42:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa946473640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa94015e200 con 0x7fa94006b430 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.967+0000 7fa944a10640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa94015d640 0x7fa940160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93f7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9401606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa944a10640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa94015d640 0x7fa940160bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36848/0 (socket says 172.21.15.6:36848) 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93ffff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa94006b430 0x7fa9400ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa944a10640 1 -- 172.21.15.6:0/553138135 learned_addr learned my addr 172.21.15.6:0/553138135 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93f7fe640 1 -- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa94006b430 msgr2=0x7fa9400ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93f7fe640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa94006b430 0x7fa9400ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93f7fe640 1 -- 172.21.15.6:0/553138135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa94015d640 msgr2=0x7fa940160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93f7fe640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa94015d640 0x7fa940160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93f7fe640 1 -- 172.21.15.6:0/553138135 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9400ca780 con 0x7fa94006b810 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa944a10640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa94015d640 0x7fa940160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93ffff640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa94006b430 0x7fa9400ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93f7fe640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9401606b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa930033000 tx=0x7fa9300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93cff9640 1 -- 172.21.15.6:0/553138135 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930034040 con 0x7fa94006b810 2024-03-18T14:42:26.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93cff9640 1 -- 172.21.15.6:0/553138135 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa93003aaa0 con 0x7fa94006b810 2024-03-18T14:42:26.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9400b4420 con 0x7fa94006b810 2024-03-18T14:42:26.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa93cff9640 1 -- 172.21.15.6:0/553138135 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9300443f0 con 0x7fa94006b810 2024-03-18T14:42:26.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.968+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9400c8c30 con 0x7fa94006b810 2024-03-18T14:42:26.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.969+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9400751b0 con 0x7fa94006b810 2024-03-18T14:42:26.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.970+0000 7fa93cff9640 1 -- 172.21.15.6:0/553138135 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa93004b060 con 0x7fa94006b810 2024-03-18T14:42:26.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.970+0000 7fa93cff9640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa92c06bbd0 0x7fa92c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:26.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.970+0000 7fa93cff9640 1 -- 172.21.15.6:0/553138135 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa9300aca90 con 0x7fa94006b810 2024-03-18T14:42:26.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.970+0000 7fa93ffff640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa92c06bbd0 0x7fa92c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:26.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.971+0000 7fa93ffff640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa92c06bbd0 0x7fa92c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa934000f60 tx=0x7fa934007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:26.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:26.979+0000 7fa93cff9640 1 -- 172.21.15.6:0/553138135 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa9400ca780 con 0x7fa94006b810 2024-03-18T14:42:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.165+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa940073d00 con 0x7fa94006b810 2024-03-18T14:42:27.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.165+0000 7fa93cff9640 1 -- 172.21.15.6:0/553138135 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa93003f070 con 0x7fa94006b810 2024-03-18T14:42:27.168 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa92c06bbd0 msgr2=0x7fa92c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa92c06bbd0 0x7fa92c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa934000f60 tx=0x7fa934007d50 comp rx=0 tx=0).stop 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 msgr2=0x7fa9401606b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9401606b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa930033000 tx=0x7fa9300046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 shutdown_connections 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa92c06bbd0 0x7fa92c06e090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa94015d640 0x7fa940160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa94006b810 0x7fa9401606b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 --2- 172.21.15.6:0/553138135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa94006b430 0x7fa9400ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.167+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 >> 172.21.15.6:0/553138135 conn(0x7fa940099a50 msgr2=0x7fa9400cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.168+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 shutdown_connections 2024-03-18T14:42:27.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.168+0000 7fa946473640 1 -- 172.21.15.6:0/553138135 wait complete. 2024-03-18T14:42:27.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.283+0000 7f3b58977640 1 Processor -- start 2024-03-18T14:42:27.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.283+0000 7f3b58977640 1 -- start start 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b58977640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b58977640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b540add30 0x7f3b5415e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b58977640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b540aff50 con 0x7f3b540b0f30 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b58977640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3b540b0560 con 0x7f3b540add30 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b58977640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3b540b1f60 con 0x7f3b5415dfb0 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b52d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b540add30 0x7f3b5415e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b5259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b4bfff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3b540b0f30 0x7f3b540ad620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:50676/0 (socket says 172.21.15.6:50676) 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b52d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b540add30 0x7f3b5415e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36850/0 (socket says 172.21.15.6:36850) 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b4bfff640 1 -- 172.21.15.6:0/1870316220 learned_addr learned my addr 172.21.15.6:0/1870316220 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b51d9b640 1 -- 172.21.15.6:0/1870316220 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778537373 0 0) 0x7f3b540aff50 con 0x7f3b540b0f30 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b51d9b640 1 -- 172.21.15.6:0/1870316220 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b40003520 con 0x7f3b540b0f30 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b5259c640 1 -- 172.21.15.6:0/1870316220 >> v2:172.21.15.6:3300/0 conn(0x7f3b540add30 msgr2=0x7f3b5415e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b5259c640 1 --2- 172.21.15.6:0/1870316220 >> v2:172.21.15.6:3300/0 conn(0x7f3b540add30 0x7f3b5415e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b5259c640 1 -- 172.21.15.6:0/1870316220 >> v1:172.21.15.92:6789/0 conn(0x7f3b540b0f30 legacy=0x7f3b540ad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b5259c640 1 -- 172.21.15.6:0/1870316220 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b540b01f0 con 0x7f3b5415dfb0 2024-03-18T14:42:27.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.284+0000 7f3b52d9d640 1 --2- 172.21.15.6:0/1870316220 >> v2:172.21.15.6:3300/0 conn(0x7f3b540add30 0x7f3b5415e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:27.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.285+0000 7f3b5259c640 1 --2- 172.21.15.6:0/1870316220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540b09f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b440097a0 tx=0x7f3b4402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32e611d271a2e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.285+0000 7f3b51d9b640 1 -- 172.21.15.6:0/1870316220 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44035020 con 0x7f3b5415dfb0 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.285+0000 7f3b51d9b640 1 -- 172.21.15.6:0/1870316220 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b44031830 con 0x7f3b5415dfb0 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.285+0000 7f3b51d9b640 1 -- 172.21.15.6:0/1870316220 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44031be0 con 0x7f3b5415dfb0 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 -- 172.21.15.6:0/1870316220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 msgr2=0x7f3b540b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 --2- 172.21.15.6:0/1870316220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540b09f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b440097a0 tx=0x7f3b4402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 -- 172.21.15.6:0/1870316220 shutdown_connections 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 --2- 172.21.15.6:0/1870316220 >> v2:172.21.15.6:3300/0 conn(0x7f3b540add30 0x7f3b5415e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 --2- 172.21.15.6:0/1870316220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540b09f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 -- 172.21.15.6:0/1870316220 >> 172.21.15.6:0/1870316220 conn(0x7f3b540d09b0 msgr2=0x7f3b540d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 -- 172.21.15.6:0/1870316220 shutdown_connections 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.286+0000 7f3b58977640 1 -- 172.21.15.6:0/1870316220 wait complete. 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.287+0000 7f3b58977640 1 Processor -- start 2024-03-18T14:42:27.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.287+0000 7f3b58977640 1 -- start start 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b58977640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540b0f30 0x7f3b540b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b58977640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b58977640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540cb520 0x7f3b540cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b58977640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540afc50 con 0x7f3b540cb520 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b58977640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540afad0 con 0x7f3b540b0f30 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b58977640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3b540b1b10 con 0x7f3b5415dfb0 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540cb520 0x7f3b540cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540cb520 0x7f3b540cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36852/0 (socket says 172.21.15.6:36852) 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 -- 172.21.15.6:0/3143150329 learned_addr learned my addr 172.21.15.6:0/3143150329 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b5259c640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540b0f30 0x7f3b540b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b4bfff640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 -- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 msgr2=0x7f3b540cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 -- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540b0f30 msgr2=0x7f3b540b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540b0f30 0x7f3b540b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.288+0000 7f3b52d9d640 1 -- 172.21.15.6:0/3143150329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b540c4bf0 con 0x7f3b540cb520 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.289+0000 7f3b52d9d640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540cb520 0x7f3b540cd340 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00efb0 tx=0x7f3b4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.289+0000 7f3b4affd640 1 -- 172.21.15.6:0/3143150329 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c018020 con 0x7f3b540cb520 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.289+0000 7f3b4affd640 1 -- 172.21.15.6:0/3143150329 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4c014480 con 0x7f3b540cb520 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.289+0000 7f3b4affd640 1 -- 172.21.15.6:0/3143150329 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c014850 con 0x7f3b540cb520 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.289+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b540c4ed0 con 0x7f3b540cb520 2024-03-18T14:42:27.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.289+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b540c5490 con 0x7f3b540cb520 2024-03-18T14:42:27.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.290+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b540b09f0 con 0x7f3b540cb520 2024-03-18T14:42:27.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.290+0000 7f3b4affd640 1 -- 172.21.15.6:0/3143150329 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3b4c005d10 con 0x7f3b540cb520 2024-03-18T14:42:27.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.291+0000 7f3b4affd640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3b3406bbd0 0x7f3b3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.291+0000 7f3b4affd640 1 -- 172.21.15.6:0/3143150329 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3b4c08bbc0 con 0x7f3b540cb520 2024-03-18T14:42:27.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.291+0000 7f3b5259c640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3b3406bbd0 0x7f3b3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.291+0000 7f3b5259c640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3b3406bbd0 0x7f3b3406e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3b44036000 tx=0x7f3b4403c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.295+0000 7f3b4affd640 1 -- 172.21.15.6:0/3143150329 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3b540b09f0 con 0x7f3b540cb520 2024-03-18T14:42:27.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.476+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3b540751b0 con 0x7f3b540cb520 2024-03-18T14:42:27.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.477+0000 7f3b4affd640 1 -- 172.21.15.6:0/3143150329 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f3b4c022030 con 0x7f3b540cb520 2024-03-18T14:42:27.479 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:27.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.478+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3b3406bbd0 msgr2=0x7f3b3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.478+0000 7f3b58977640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3b3406bbd0 0x7f3b3406e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3b44036000 tx=0x7f3b4403c040 comp rx=0 tx=0).stop 2024-03-18T14:42:27.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.478+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540cb520 msgr2=0x7f3b540cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.478+0000 7f3b58977640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540cb520 0x7f3b540cd340 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00efb0 tx=0x7f3b4c00c490 comp rx=0 tx=0).stop 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 shutdown_connections 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3b3406bbd0 0x7f3b3406e090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540cb520 0x7f3b540cd340 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5415dfb0 0x7f3b540cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 --2- 172.21.15.6:0/3143150329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540b0f30 0x7f3b540b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 >> 172.21.15.6:0/3143150329 conn(0x7f3b540d09b0 msgr2=0x7f3b540ad620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 shutdown_connections 2024-03-18T14:42:27.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.479+0000 7f3b58977640 1 -- 172.21.15.6:0/3143150329 wait complete. 2024-03-18T14:42:27.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:27.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.594+0000 7fccb1587640 1 Processor -- start 2024-03-18T14:42:27.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.594+0000 7fccb1587640 1 -- start start 2024-03-18T14:42:27.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccb1587640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fccac143ed0 0x7fccac0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccb1587640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac1443c0 0x7fccac0b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccb1587640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fccac0b0cd0 0x7fccac071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccb1587640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fccac0cf7d0 con 0x7fccac143ed0 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccb1587640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fccac15f230 con 0x7fccac0b0cd0 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccb1587640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fccac15f6c0 con 0x7fccac1443c0 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccaaffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fccac143ed0 0x7fccac0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccaaffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fccac143ed0 0x7fccac0cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36868/0 (socket says 172.21.15.6:36868) 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccaaffd640 1 -- 172.21.15.6:0/498012469 learned_addr learned my addr 172.21.15.6:0/498012469 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fccab7fe640 1 --2- 172.21.15.6:0/498012469 >> v2:172.21.15.92:3300/0 conn(0x7fccac0b0cd0 0x7fccac071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.595+0000 7fcca27fc640 1 --2- 172.21.15.6:0/498012469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac1443c0 0x7fccac0b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.596+0000 7fccaaffd640 1 -- 172.21.15.6:0/498012469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac1443c0 msgr2=0x7fccac0b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.596+0000 7fccaaffd640 1 --2- 172.21.15.6:0/498012469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac1443c0 0x7fccac0b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.596+0000 7fccaaffd640 1 -- 172.21.15.6:0/498012469 >> v2:172.21.15.92:3300/0 conn(0x7fccac0b0cd0 msgr2=0x7fccac071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:27.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.596+0000 7fccaaffd640 1 --2- 172.21.15.6:0/498012469 >> v2:172.21.15.92:3300/0 conn(0x7fccac0b0cd0 0x7fccac071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.596+0000 7fccaaffd640 1 -- 172.21.15.6:0/498012469 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccac15e900 con 0x7fccac143ed0 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.596+0000 7fccaaffd640 1 --2- 172.21.15.6:0/498012469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac143ed0 0x7fccac0cee80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fccac0ce910 tx=0x7fcc9c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a8c8200f358f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccaa7fc640 1 -- 172.21.15.6:0/498012469 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c03b070 con 0x7fccac143ed0 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccaa7fc640 1 -- 172.21.15.6:0/498012469 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc9c036440 con 0x7fccac143ed0 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccaa7fc640 1 -- 172.21.15.6:0/498012469 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c036810 con 0x7fccac143ed0 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 -- 172.21.15.6:0/498012469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac143ed0 msgr2=0x7fccac0cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 --2- 172.21.15.6:0/498012469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac143ed0 0x7fccac0cee80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fccac0ce910 tx=0x7fcc9c02e0e0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 -- 172.21.15.6:0/498012469 shutdown_connections 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 --2- 172.21.15.6:0/498012469 >> v2:172.21.15.92:3300/0 conn(0x7fccac0b0cd0 0x7fccac071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 --2- 172.21.15.6:0/498012469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac1443c0 0x7fccac0b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 --2- 172.21.15.6:0/498012469 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac143ed0 0x7fccac0cee80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 -- 172.21.15.6:0/498012469 >> 172.21.15.6:0/498012469 conn(0x7fccac06b1c0 msgr2=0x7fccac06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:27.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 -- 172.21.15.6:0/498012469 shutdown_connections 2024-03-18T14:42:27.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.597+0000 7fccb1587640 1 -- 172.21.15.6:0/498012469 wait complete. 2024-03-18T14:42:27.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.598+0000 7fccb1587640 1 Processor -- start 2024-03-18T14:42:27.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccb1587640 1 -- start start 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccb1587640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccac0b0cd0 0x7fccac14be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccb1587640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac143ed0 0x7fccac14aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccb1587640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac1443c0 0x7fccac13d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccb1587640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fccac0cf3c0 con 0x7fccac1443c0 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccb1587640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fccac15eec0 con 0x7fccac0b0cd0 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccb1587640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fccac0cf540 con 0x7fccac143ed0 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccab7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac1443c0 0x7fccac13d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccab7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac1443c0 0x7fccac13d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36884/0 (socket says 172.21.15.6:36884) 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccab7fe640 1 -- 172.21.15.6:0/2559512690 learned_addr learned my addr 172.21.15.6:0/2559512690 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fccaaffd640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccac0b0cd0 0x7fccac14be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.599+0000 7fcca27fc640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac143ed0 0x7fccac14aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.600+0000 7fccab7fe640 1 -- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac143ed0 msgr2=0x7fccac14aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.600+0000 7fccab7fe640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac143ed0 0x7fccac14aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.600+0000 7fccab7fe640 1 -- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccac0b0cd0 msgr2=0x7fccac14be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.600+0000 7fccab7fe640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccac0b0cd0 0x7fccac14be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.600+0000 7fccab7fe640 1 -- 172.21.15.6:0/2559512690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccac14c390 con 0x7fccac1443c0 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.600+0000 7fccab7fe640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac1443c0 0x7fccac13d750 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcca400ef00 tx=0x7fcca400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.601+0000 7fcca3fff640 1 -- 172.21.15.6:0/2559512690 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4018020 con 0x7fccac1443c0 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.601+0000 7fcca3fff640 1 -- 172.21.15.6:0/2559512690 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca400bd20 con 0x7fccac1443c0 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.601+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccac149960 con 0x7fccac1443c0 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.601+0000 7fcca3fff640 1 -- 172.21.15.6:0/2559512690 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4014640 con 0x7fccac1443c0 2024-03-18T14:42:27.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.601+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccac149f20 con 0x7fccac1443c0 2024-03-18T14:42:27.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.602+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccac0cea60 con 0x7fccac1443c0 2024-03-18T14:42:27.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.603+0000 7fcca3fff640 1 -- 172.21.15.6:0/2559512690 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcca4005d10 con 0x7fccac1443c0 2024-03-18T14:42:27.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.603+0000 7fcca3fff640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc8c06bc20 0x7fcc8c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.603+0000 7fcca3fff640 1 -- 172.21.15.6:0/2559512690 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcca408b2a0 con 0x7fccac1443c0 2024-03-18T14:42:27.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.603+0000 7fccaaffd640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc8c06bc20 0x7fcc8c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.604+0000 7fccaaffd640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc8c06bc20 0x7fcc8c06e0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c038000 tx=0x7fcc9c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.608+0000 7fcca3fff640 1 -- 172.21.15.6:0/2559512690 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fccac0cea60 con 0x7fccac1443c0 2024-03-18T14:42:27.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.789+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fccac0751b0 con 0x7fccac1443c0 2024-03-18T14:42:27.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.789+0000 7fcca3fff640 1 -- 172.21.15.6:0/2559512690 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fcca4022070 con 0x7fccac1443c0 2024-03-18T14:42:27.792 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc8c06bc20 msgr2=0x7fcc8c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc8c06bc20 0x7fcc8c06e0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c038000 tx=0x7fcc9c032000 comp rx=0 tx=0).stop 2024-03-18T14:42:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac1443c0 msgr2=0x7fccac13d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac1443c0 0x7fccac13d750 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcca400ef00 tx=0x7fcca400b480 comp rx=0 tx=0).stop 2024-03-18T14:42:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 shutdown_connections 2024-03-18T14:42:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc8c06bc20 0x7fcc8c06e0e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccac1443c0 0x7fccac13d750 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccac143ed0 0x7fccac14aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 --2- 172.21.15.6:0/2559512690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccac0b0cd0 0x7fccac14be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 >> 172.21.15.6:0/2559512690 conn(0x7fccac06b1c0 msgr2=0x7fccac0cd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 shutdown_connections 2024-03-18T14:42:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.791+0000 7fccb1587640 1 -- 172.21.15.6:0/2559512690 wait complete. 2024-03-18T14:42:27.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.904+0000 7fe993d89640 1 Processor -- start 2024-03-18T14:42:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.904+0000 7fe993d89640 1 -- start start 2024-03-18T14:42:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe993d89640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe98c06c7a0 0x7fe98c0d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe993d89640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c0b6df0 0x7fe98c0b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe993d89640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c0d1f30 con 0x7fe98c06cb80 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe993d89640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe98c0b9cf0 con 0x7fe98c06c7a0 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe993d89640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe98c0d2210 con 0x7fe98c0b6df0 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe991324640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe98c06cb80 0x7fe98c0b95e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55958/0 (socket says 172.21.15.6:55958) 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe991324640 1 -- 172.21.15.6:0/1682404489 learned_addr learned my addr 172.21.15.6:0/1682404489 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe992326640 1 --2- 172.21.15.6:0/1682404489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c0b6df0 0x7fe98c0b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe991b25640 1 --2- 172.21.15.6:0/1682404489 >> v2:172.21.15.92:3300/0 conn(0x7fe98c06c7a0 0x7fe98c0d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.905+0000 7fe990b23640 1 -- 172.21.15.6:0/1682404489 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1306120598 0 0) 0x7fe98c0d1f30 con 0x7fe98c06cb80 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.906+0000 7fe990b23640 1 -- 172.21.15.6:0/1682404489 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe978003520 con 0x7fe98c06cb80 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.906+0000 7fe992326640 1 -- 172.21.15.6:0/1682404489 >> v2:172.21.15.92:3300/0 conn(0x7fe98c06c7a0 msgr2=0x7fe98c0d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.906+0000 7fe992326640 1 --2- 172.21.15.6:0/1682404489 >> v2:172.21.15.92:3300/0 conn(0x7fe98c06c7a0 0x7fe98c0d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.906+0000 7fe992326640 1 -- 172.21.15.6:0/1682404489 >> v1:172.21.15.6:6789/0 conn(0x7fe98c06cb80 legacy=0x7fe98c0b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.906+0000 7fe992326640 1 -- 172.21.15.6:0/1682404489 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98c0d24f0 con 0x7fe98c0b6df0 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.906+0000 7fe992326640 1 --2- 172.21.15.6:0/1682404489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c0b6df0 0x7fe98c0b7280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe988009d90 tx=0x7fe98802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4834c645e0f4f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe990b23640 1 -- 172.21.15.6:0/1682404489 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988039070 con 0x7fe98c0b6df0 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe990b23640 1 -- 172.21.15.6:0/1682404489 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98802cce0 con 0x7fe98c0b6df0 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe990b23640 1 -- 172.21.15.6:0/1682404489 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988034660 con 0x7fe98c0b6df0 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 -- 172.21.15.6:0/1682404489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c0b6df0 msgr2=0x7fe98c0b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 --2- 172.21.15.6:0/1682404489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c0b6df0 0x7fe98c0b7280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe988009d90 tx=0x7fe98802bf30 comp rx=0 tx=0).stop 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 -- 172.21.15.6:0/1682404489 shutdown_connections 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 --2- 172.21.15.6:0/1682404489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c0b6df0 0x7fe98c0b7280 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 --2- 172.21.15.6:0/1682404489 >> v2:172.21.15.92:3300/0 conn(0x7fe98c06c7a0 0x7fe98c0d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 -- 172.21.15.6:0/1682404489 >> 172.21.15.6:0/1682404489 conn(0x7fe98c072ce0 msgr2=0x7fe98c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:27.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 -- 172.21.15.6:0/1682404489 shutdown_connections 2024-03-18T14:42:27.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.907+0000 7fe993d89640 1 -- 172.21.15.6:0/1682404489 wait complete. 2024-03-18T14:42:27.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.908+0000 7fe993d89640 1 Processor -- start 2024-03-18T14:42:27.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.908+0000 7fe993d89640 1 -- start start 2024-03-18T14:42:27.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe993d89640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe98c06c7a0 0x7fe98c0abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe993d89640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c06cb80 0x7fe98c160740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe993d89640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe98c0b6df0 0x7fe98c1f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe993d89640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c06cfe0 con 0x7fe98c0b6df0 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe993d89640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c0d2ab0 con 0x7fe98c06c7a0 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe993d89640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe98c0d37f0 con 0x7fe98c06cb80 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe992326640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe98c0b6df0 0x7fe98c1f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe992326640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe98c0b6df0 0x7fe98c1f38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36888/0 (socket says 172.21.15.6:36888) 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe992326640 1 -- 172.21.15.6:0/3043332255 learned_addr learned my addr 172.21.15.6:0/3043332255 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe991b25640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe98c06c7a0 0x7fe98c0abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.909+0000 7fe991324640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c06cb80 0x7fe98c160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe992326640 1 -- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c06cb80 msgr2=0x7fe98c160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe992326640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c06cb80 0x7fe98c160740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe992326640 1 -- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe98c06c7a0 msgr2=0x7fe98c0abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe992326640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe98c06c7a0 0x7fe98c0abd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:27.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe992326640 1 -- 172.21.15.6:0/3043332255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98c0aa6f0 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe992326640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe98c0b6df0 0x7fe98c1f38e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe988009d90 tx=0x7fe988004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe9827fc640 1 -- 172.21.15.6:0/3043332255 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988039050 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe9827fc640 1 -- 172.21.15.6:0/3043332255 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe988045070 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe98c0aa9d0 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe9827fc640 1 -- 172.21.15.6:0/3043332255 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988009450 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.910+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe98c0aaf90 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.911+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe98c06d200 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.911+0000 7fe9827fc640 1 -- 172.21.15.6:0/3043332255 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe98803e850 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.912+0000 7fe9827fc640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe97006bbd0 0x7fe97006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.912+0000 7fe9827fc640 1 -- 172.21.15.6:0/3043332255 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe9880ad9d0 con 0x7fe98c0b6df0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.912+0000 7fe991b25640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe97006bbd0 0x7fe97006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:27.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.912+0000 7fe991b25640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe97006bbd0 0x7fe97006e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe98c15c260 tx=0x7fe984009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:27.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:27.916+0000 7fe9827fc640 1 -- 172.21.15.6:0/3043332255 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe98c06d200 con 0x7fe98c0b6df0 2024-03-18T14:42:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.112+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe98c06b7f0 con 0x7fe98c0b6df0 2024-03-18T14:42:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.113+0000 7fe9827fc640 1 -- 172.21.15.6:0/3043332255 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fe98807bc20 con 0x7fe98c0b6df0 2024-03-18T14:42:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:28.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe97006bbd0 msgr2=0x7fe97006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe97006bbd0 0x7fe97006e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe98c15c260 tx=0x7fe984009190 comp rx=0 tx=0).stop 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe98c0b6df0 msgr2=0x7fe98c1f38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe98c0b6df0 0x7fe98c1f38e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe988009d90 tx=0x7fe988004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 shutdown_connections 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe97006bbd0 0x7fe97006e090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe98c0b6df0 0x7fe98c1f38e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe98c06cb80 0x7fe98c160740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 --2- 172.21.15.6:0/3043332255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe98c06c7a0 0x7fe98c0abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 >> 172.21.15.6:0/3043332255 conn(0x7fe98c072ce0 msgr2=0x7fe98c0cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 shutdown_connections 2024-03-18T14:42:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.115+0000 7fe993d89640 1 -- 172.21.15.6:0/3043332255 wait complete. 2024-03-18T14:42:28.128 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:42:28.128 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:28.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.282+0000 7ff919444640 1 Processor -- start 2024-03-18T14:42:28.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.282+0000 7ff919444640 1 -- start start 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff919444640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff91406b8d0 0x7ff9140cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff919444640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140cd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff919444640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9140cce30 con 0x7ff91415e240 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff919444640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff9140cb000 con 0x7ff91406b8d0 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff919444640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff9140b1f90 con 0x7ff9140afd00 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff9127fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff91406b8d0 0x7ff9140cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff912ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff91415e240 0x7ff91406b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55964/0 (socket says 172.21.15.6:55964) 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff912ffd640 1 -- 172.21.15.6:0/109786076 learned_addr learned my addr 172.21.15.6:0/109786076 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:28.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff9137fe640 1 --2- 172.21.15.6:0/109786076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff9127fc640 1 -- 172.21.15.6:0/109786076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 msgr2=0x7ff9140cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff9127fc640 1 --2- 172.21.15.6:0/109786076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff9127fc640 1 -- 172.21.15.6:0/109786076 >> v1:172.21.15.6:6789/0 conn(0x7ff91415e240 legacy=0x7ff91406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.283+0000 7ff9127fc640 1 -- 172.21.15.6:0/109786076 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9140b1c20 con 0x7ff91406b8d0 2024-03-18T14:42:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.284+0000 7ff9127fc640 1 --2- 172.21.15.6:0/109786076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91406b8d0 0x7ff9140cd0d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff9140b2110 tx=0x7ff8fc02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37de957f8cd831f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.284+0000 7ff911ffb640 1 -- 172.21.15.6:0/109786076 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc031770 con 0x7ff91406b8d0 2024-03-18T14:42:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.284+0000 7ff911ffb640 1 -- 172.21.15.6:0/109786076 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8fc037070 con 0x7ff91406b8d0 2024-03-18T14:42:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.284+0000 7ff911ffb640 1 -- 172.21.15.6:0/109786076 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc031b30 con 0x7ff91406b8d0 2024-03-18T14:42:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 -- 172.21.15.6:0/109786076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91406b8d0 msgr2=0x7ff9140cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 --2- 172.21.15.6:0/109786076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91406b8d0 0x7ff9140cd0d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff9140b2110 tx=0x7ff8fc02b810 comp rx=0 tx=0).stop 2024-03-18T14:42:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 -- 172.21.15.6:0/109786076 shutdown_connections 2024-03-18T14:42:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 --2- 172.21.15.6:0/109786076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 --2- 172.21.15.6:0/109786076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91406b8d0 0x7ff9140cd0d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 -- 172.21.15.6:0/109786076 >> 172.21.15.6:0/109786076 conn(0x7ff9140d09b0 msgr2=0x7ff9140d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 -- 172.21.15.6:0/109786076 shutdown_connections 2024-03-18T14:42:28.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.285+0000 7ff919444640 1 -- 172.21.15.6:0/109786076 wait complete. 2024-03-18T14:42:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.286+0000 7ff919444640 1 Processor -- start 2024-03-18T14:42:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.286+0000 7ff919444640 1 -- start start 2024-03-18T14:42:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff919444640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff91406b8d0 0x7ff914147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff919444640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff919444640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91415e240 0x7ff914159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff919444640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff9140ce580 con 0x7ff91415e240 2024-03-18T14:42:28.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff919444640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff9140b1180 con 0x7ff91406b8d0 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff919444640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff9140ce700 con 0x7ff9140afd00 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91415e240 0x7ff914159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91415e240 0x7ff914159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36912/0 (socket says 172.21.15.6:36912) 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 -- 172.21.15.6:0/1617068645 learned_addr learned my addr 172.21.15.6:0/1617068645 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff912ffd640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff91406b8d0 0x7ff914147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9127fc640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 -- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 msgr2=0x7ff9140adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 -- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff91406b8d0 msgr2=0x7ff914147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff91406b8d0 0x7ff914147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.287+0000 7ff9137fe640 1 -- 172.21.15.6:0/1617068645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91415a3a0 con 0x7ff91415e240 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.288+0000 7ff9137fe640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91415e240 0x7ff914159e60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff90400efc0 tx=0x7ff90400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.288+0000 7ff90b7fe640 1 -- 172.21.15.6:0/1617068645 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904018020 con 0x7ff91415e240 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.288+0000 7ff90b7fe640 1 -- 172.21.15.6:0/1617068645 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff904014480 con 0x7ff91415e240 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.288+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9140ae580 con 0x7ff91415e240 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.288+0000 7ff90b7fe640 1 -- 172.21.15.6:0/1617068645 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904014870 con 0x7ff91415e240 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.288+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9140aeb40 con 0x7ff91415e240 2024-03-18T14:42:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.289+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff91409e9a0 con 0x7ff91415e240 2024-03-18T14:42:28.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.289+0000 7ff90b7fe640 1 -- 172.21.15.6:0/1617068645 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff904005d10 con 0x7ff91415e240 2024-03-18T14:42:28.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.290+0000 7ff90b7fe640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff8f406bbd0 0x7ff8f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.290+0000 7ff90b7fe640 1 -- 172.21.15.6:0/1617068645 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff90408b7f0 con 0x7ff91415e240 2024-03-18T14:42:28.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.290+0000 7ff912ffd640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff8f406bbd0 0x7ff8f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.290+0000 7ff912ffd640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff8f406bbd0 0x7ff8f406e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff9141f76e0 tx=0x7ff90c007e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.294+0000 7ff90b7fe640 1 -- 172.21.15.6:0/1617068645 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff904022070 con 0x7ff91415e240 2024-03-18T14:42:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.476+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9140b65e0 con 0x7ff91415e240 2024-03-18T14:42:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.477+0000 7ff90b7fe640 1 -- 172.21.15.6:0/1617068645 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7ff904010040 con 0x7ff91415e240 2024-03-18T14:42:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:28.481 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:28.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff8f406bbd0 msgr2=0x7ff8f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff8f406bbd0 0x7ff8f406e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff9141f76e0 tx=0x7ff90c007e80 comp rx=0 tx=0).stop 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91415e240 msgr2=0x7ff914159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91415e240 0x7ff914159e60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff90400efc0 tx=0x7ff90400c490 comp rx=0 tx=0).stop 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 shutdown_connections 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff8f406bbd0 0x7ff8f406e090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff91415e240 0x7ff914159e60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9140afd00 0x7ff9140adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 --2- 172.21.15.6:0/1617068645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff91406b8d0 0x7ff914147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.479+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 >> 172.21.15.6:0/1617068645 conn(0x7ff9140d09b0 msgr2=0x7ff91406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.480+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 shutdown_connections 2024-03-18T14:42:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.480+0000 7ff919444640 1 -- 172.21.15.6:0/1617068645 wait complete. 2024-03-18T14:42:28.493 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:28.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.595+0000 7f8266359640 1 Processor -- start 2024-03-18T14:42:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.595+0000 7f8266359640 1 -- start start 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f8266359640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f826006d440 0x7f826006c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f8266359640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f826009a3c0 0x7f826015d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f8266359640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826006d820 con 0x7f826009b070 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f8266359640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f82600d0d60 con 0x7f826009a3c0 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f8266359640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f82600d0150 con 0x7f826006d440 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f82648f6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f826009a3c0 0x7f826015d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f825f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f826009b070 0x7f8260099cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55980/0 (socket says 172.21.15.6:55980) 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f825f7fe640 1 -- 172.21.15.6:0/4224180109 learned_addr learned my addr 172.21.15.6:0/4224180109 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f825ffff640 1 --2- 172.21.15.6:0/4224180109 >> v2:172.21.15.92:3300/0 conn(0x7f826006d440 0x7f826006c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f82648f6640 1 -- 172.21.15.6:0/4224180109 >> v2:172.21.15.92:3300/0 conn(0x7f826006d440 msgr2=0x7f826006c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f82648f6640 1 --2- 172.21.15.6:0/4224180109 >> v2:172.21.15.92:3300/0 conn(0x7f826006d440 0x7f826006c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f82648f6640 1 -- 172.21.15.6:0/4224180109 >> v1:172.21.15.6:6789/0 conn(0x7f826009b070 legacy=0x7f8260099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f82648f6640 1 -- 172.21.15.6:0/4224180109 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82600d0430 con 0x7f826009a3c0 2024-03-18T14:42:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.596+0000 7f825effd640 1 -- 172.21.15.6:0/4224180109 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 275148488 0 0) 0x7f826006d820 con 0x7f826009b070 2024-03-18T14:42:28.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.597+0000 7f82648f6640 1 --2- 172.21.15.6:0/4224180109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f826015d7e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8260072c40 tx=0x7f825802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c317ed45a35b516 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.597+0000 7f825effd640 1 -- 172.21.15.6:0/4224180109 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258039070 con 0x7f826009a3c0 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.597+0000 7f825effd640 1 -- 172.21.15.6:0/4224180109 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f825802cc80 con 0x7f826009a3c0 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.597+0000 7f825effd640 1 -- 172.21.15.6:0/4224180109 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258034660 con 0x7f826009a3c0 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 -- 172.21.15.6:0/4224180109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 msgr2=0x7f826015d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 --2- 172.21.15.6:0/4224180109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f826015d7e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8260072c40 tx=0x7f825802bed0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 -- 172.21.15.6:0/4224180109 shutdown_connections 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 --2- 172.21.15.6:0/4224180109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f826015d7e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 --2- 172.21.15.6:0/4224180109 >> v2:172.21.15.92:3300/0 conn(0x7f826006d440 0x7f826006c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 -- 172.21.15.6:0/4224180109 >> 172.21.15.6:0/4224180109 conn(0x7f82600b3660 msgr2=0x7f82600b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 -- 172.21.15.6:0/4224180109 shutdown_connections 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.598+0000 7f8266359640 1 -- 172.21.15.6:0/4224180109 wait complete. 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.599+0000 7f8266359640 1 Processor -- start 2024-03-18T14:42:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.599+0000 7f8266359640 1 -- start start 2024-03-18T14:42:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f8266359640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f826006d440 0x7f8260153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f8266359640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f8260070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f8266359640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82600714c0 0x7f82600c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f8266359640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f82600cfd70 con 0x7f826009a3c0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f8266359640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f82600d09f0 con 0x7f826006d440 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f8266359640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f82600cfef0 con 0x7f82600714c0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f8260070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f8260070f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36926/0 (socket says 172.21.15.6:36926) 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 -- 172.21.15.6:0/2059731107 learned_addr learned my addr 172.21.15.6:0/2059731107 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825ffff640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f826006d440 0x7f8260153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f82648f6640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82600714c0 0x7f82600c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 -- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82600714c0 msgr2=0x7f82600c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82600714c0 0x7f82600c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 -- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f826006d440 msgr2=0x7f8260153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f826006d440 0x7f8260153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 -- 172.21.15.6:0/2059731107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82600c7c20 con 0x7f826009a3c0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.600+0000 7f825f7fe640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f8260070f80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f825000c430 tx=0x7f825000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.601+0000 7f825cff9640 1 -- 172.21.15.6:0/2059731107 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82500074e0 con 0x7f826009a3c0 2024-03-18T14:42:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.601+0000 7f825cff9640 1 -- 172.21.15.6:0/2059731107 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82500047e0 con 0x7f826009a3c0 2024-03-18T14:42:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.601+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82600c7f00 con 0x7f826009a3c0 2024-03-18T14:42:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.601+0000 7f825cff9640 1 -- 172.21.15.6:0/2059731107 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250004bd0 con 0x7f826009a3c0 2024-03-18T14:42:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.601+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826009b910 con 0x7f826009a3c0 2024-03-18T14:42:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.602+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82600751b0 con 0x7f826009a3c0 2024-03-18T14:42:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.602+0000 7f825cff9640 1 -- 172.21.15.6:0/2059731107 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8250011070 con 0x7f826009a3c0 2024-03-18T14:42:28.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.602+0000 7f825cff9640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f824c06bbd0 0x7f824c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.602+0000 7f825cff9640 1 -- 172.21.15.6:0/2059731107 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f825008a910 con 0x7f826009a3c0 2024-03-18T14:42:28.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.602+0000 7f825ffff640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f824c06bbd0 0x7f824c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.603+0000 7f825ffff640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f824c06bbd0 0x7f824c06e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8254009b40 tx=0x7f8254009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.607+0000 7f825cff9640 1 -- 172.21.15.6:0/2059731107 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8250058a40 con 0x7f826009a3c0 2024-03-18T14:42:28.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.787+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f826009a7e0 con 0x7f826009a3c0 2024-03-18T14:42:28.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.788+0000 7f825cff9640 1 -- 172.21.15.6:0/2059731107 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f825005cae0 con 0x7f826009a3c0 2024-03-18T14:42:28.791 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f824c06bbd0 msgr2=0x7f824c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f824c06bbd0 0x7f824c06e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8254009b40 tx=0x7f8254009190 comp rx=0 tx=0).stop 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 msgr2=0x7f8260070f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f8260070f80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f825000c430 tx=0x7f825000c8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 shutdown_connections 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f824c06bbd0 0x7f824c06e090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82600714c0 0x7f82600c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f826009a3c0 0x7f8260070f80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 --2- 172.21.15.6:0/2059731107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f826006d440 0x7f8260153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 >> 172.21.15.6:0/2059731107 conn(0x7f82600b3660 msgr2=0x7f82600cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 shutdown_connections 2024-03-18T14:42:28.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.790+0000 7f8266359640 1 -- 172.21.15.6:0/2059731107 wait complete. 2024-03-18T14:42:28.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.905+0000 7f1987e75640 1 Processor -- start 2024-03-18T14:42:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.905+0000 7f1987e75640 1 -- start start 2024-03-18T14:42:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1987e75640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19800d0730 0x7f19800cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1987e75640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d2510 0x7f198015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1987e75640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19800d2940 con 0x7f19800ce9d0 2024-03-18T14:42:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1987e75640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f19800cd8d0 con 0x7f19800d0730 2024-03-18T14:42:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1987e75640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f19800cfac0 con 0x7f19800d2510 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1985410640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19800d0730 0x7f19800cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1985c11640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19800ce9d0 0x7f19800d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55986/0 (socket says 172.21.15.6:55986) 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1985c11640 1 -- 172.21.15.6:0/4189927586 learned_addr learned my addr 172.21.15.6:0/4189927586 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.906+0000 7f1986412640 1 --2- 172.21.15.6:0/4189927586 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d2510 0x7f198015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.907+0000 7f1985410640 1 -- 172.21.15.6:0/4189927586 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d2510 msgr2=0x7f198015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.907+0000 7f1985410640 1 --2- 172.21.15.6:0/4189927586 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d2510 0x7f198015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.907+0000 7f1985410640 1 -- 172.21.15.6:0/4189927586 >> v1:172.21.15.6:6789/0 conn(0x7f19800ce9d0 legacy=0x7f19800d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.907+0000 7f1985410640 1 -- 172.21.15.6:0/4189927586 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19800cccc0 con 0x7f19800d0730 2024-03-18T14:42:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.907+0000 7f1984c0f640 1 -- 172.21.15.6:0/4189927586 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 907840457 0 0) 0x7f19800d2940 con 0x7f19800ce9d0 2024-03-18T14:42:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.907+0000 7f1985410640 1 --2- 172.21.15.6:0/4189927586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800d0730 0x7f19800cc650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f19740097a0 tx=0x7f197402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3502ec11d1a62ce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1984c0f640 1 -- 172.21.15.6:0/4189927586 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19740315a0 con 0x7f19800d0730 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1984c0f640 1 -- 172.21.15.6:0/4189927586 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1974037070 con 0x7f19800d0730 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1984c0f640 1 -- 172.21.15.6:0/4189927586 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974031960 con 0x7f19800d0730 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 -- 172.21.15.6:0/4189927586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800d0730 msgr2=0x7f19800cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 --2- 172.21.15.6:0/4189927586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800d0730 0x7f19800cc650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f19740097a0 tx=0x7f197402b840 comp rx=0 tx=0).stop 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 -- 172.21.15.6:0/4189927586 shutdown_connections 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 --2- 172.21.15.6:0/4189927586 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d2510 0x7f198015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 --2- 172.21.15.6:0/4189927586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800d0730 0x7f19800cc650 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 -- 172.21.15.6:0/4189927586 >> 172.21.15.6:0/4189927586 conn(0x7f198008b0c0 msgr2=0x7f19800d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 -- 172.21.15.6:0/4189927586 shutdown_connections 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.908+0000 7f1987e75640 1 -- 172.21.15.6:0/4189927586 wait complete. 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.909+0000 7f1987e75640 1 Processor -- start 2024-03-18T14:42:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.909+0000 7f1987e75640 1 -- start start 2024-03-18T14:42:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1987e75640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800ce9d0 0x7f1980156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1987e75640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d0730 0x7f19801583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1987e75640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19800d2510 0x7f1980148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1987e75640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f19800cd560 con 0x7f19800ce9d0 2024-03-18T14:42:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1987e75640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19800cd280 con 0x7f19800d2510 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1987e75640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f19800cedb0 con 0x7f19800d0730 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985c11640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800ce9d0 0x7f1980156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985c11640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800ce9d0 0x7f1980156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36944/0 (socket says 172.21.15.6:36944) 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985c11640 1 -- 172.21.15.6:0/605008965 learned_addr learned my addr 172.21.15.6:0/605008965 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985410640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d0730 0x7f19801583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1986412640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19800d2510 0x7f1980148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985c11640 1 -- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d0730 msgr2=0x7f19801583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985c11640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d0730 0x7f19801583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985c11640 1 -- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19800d2510 msgr2=0x7f1980148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.910+0000 7f1985c11640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19800d2510 0x7f1980148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.911+0000 7f1985c11640 1 -- 172.21.15.6:0/605008965 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19800b52b0 con 0x7f19800ce9d0 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.911+0000 7f1985c11640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800ce9d0 0x7f1980156580 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f197800c3c0 tx=0x7f197800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.911+0000 7f197e7fc640 1 -- 172.21.15.6:0/605008965 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978017070 con 0x7f19800ce9d0 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.911+0000 7f197e7fc640 1 -- 172.21.15.6:0/605008965 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19780074e0 con 0x7f19800ce9d0 2024-03-18T14:42:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.911+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19800b5550 con 0x7f19800ce9d0 2024-03-18T14:42:28.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.911+0000 7f197e7fc640 1 -- 172.21.15.6:0/605008965 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19780059e0 con 0x7f19800ce9d0 2024-03-18T14:42:28.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.911+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19800b5b10 con 0x7f19800ce9d0 2024-03-18T14:42:28.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.912+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19800751b0 con 0x7f19800ce9d0 2024-03-18T14:42:28.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.912+0000 7f197e7fc640 1 -- 172.21.15.6:0/605008965 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f197801a070 con 0x7f19800ce9d0 2024-03-18T14:42:28.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.913+0000 7f197e7fc640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f196c06bbd0 0x7f196c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.913+0000 7f197e7fc640 1 -- 172.21.15.6:0/605008965 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f197808a130 con 0x7f19800ce9d0 2024-03-18T14:42:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.913+0000 7f1985410640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f196c06bbd0 0x7f196c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.913+0000 7f1985410640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f196c06bbd0 0x7f196c06e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1974036000 tx=0x7f1974008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:28.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:28.917+0000 7f197e7fc640 1 -- 172.21.15.6:0/605008965 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1978058260 con 0x7f19800ce9d0 2024-03-18T14:42:29.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.096+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19800cfe00 con 0x7f19800ce9d0 2024-03-18T14:42:29.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.096+0000 7f197e7fc640 1 -- 172.21.15.6:0/605008965 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f197805c300 con 0x7f19800ce9d0 2024-03-18T14:42:29.099 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f196c06bbd0 msgr2=0x7f196c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f196c06bbd0 0x7f196c06e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1974036000 tx=0x7f1974008030 comp rx=0 tx=0).stop 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800ce9d0 msgr2=0x7f1980156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800ce9d0 0x7f1980156580 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f197800c3c0 tx=0x7f197800c880 comp rx=0 tx=0).stop 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 shutdown_connections 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f196c06bbd0 0x7f196c06e090 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19800d2510 0x7f1980148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19800d0730 0x7f19801583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 --2- 172.21.15.6:0/605008965 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19800ce9d0 0x7f1980156580 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 >> 172.21.15.6:0/605008965 conn(0x7f198008b0c0 msgr2=0x7f19800b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:29.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 shutdown_connections 2024-03-18T14:42:29.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.098+0000 7f1987e75640 1 -- 172.21.15.6:0/605008965 wait complete. 2024-03-18T14:42:29.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:29.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.213+0000 7f06c5259640 1 Processor -- start 2024-03-18T14:42:29.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.213+0000 7f06c5259640 1 -- start start 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06c5259640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f06c00d3570 0x7f06c00b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06c5259640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c00d0450 0x7f06c00d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06c5259640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c00d39d0 con 0x7f06c015e030 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06c5259640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f06c015e410 con 0x7f06c00d3570 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06c5259640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f06c00ce300 con 0x7f06c00d0450 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06be59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f06c00d3570 0x7f06c00b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06bed9d640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f06c015e030 0x7f06c00ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:50688/0 (socket says 172.21.15.6:50688) 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06bed9d640 1 -- 172.21.15.6:0/2426858769 learned_addr learned my addr 172.21.15.6:0/2426858769 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06bf59e640 1 --2- 172.21.15.6:0/2426858769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c00d0450 0x7f06c00d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06bdd9b640 1 -- 172.21.15.6:0/2426858769 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1640830410 0 0) 0x7f06c00d39d0 con 0x7f06c015e030 2024-03-18T14:42:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06be59c640 1 -- 172.21.15.6:0/2426858769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c00d0450 msgr2=0x7f06c00d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06be59c640 1 --2- 172.21.15.6:0/2426858769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c00d0450 0x7f06c00d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06be59c640 1 -- 172.21.15.6:0/2426858769 >> v1:172.21.15.92:6789/0 conn(0x7f06c015e030 legacy=0x7f06c00ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.214+0000 7f06be59c640 1 -- 172.21.15.6:0/2426858769 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c00d0e20 con 0x7f06c00d3570 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.215+0000 7f06bf59e640 1 --2- 172.21.15.6:0/2426858769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c00d0450 0x7f06c00d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.215+0000 7f06be59c640 1 --2- 172.21.15.6:0/2426858769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d3570 0x7f06c00b6140 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f06b40097a0 tx=0x7f06b402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0eaaf319b7fab0f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.215+0000 7f06bdd9b640 1 -- 172.21.15.6:0/2426858769 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4035020 con 0x7f06c00d3570 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.215+0000 7f06bdd9b640 1 -- 172.21.15.6:0/2426858769 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b4031810 con 0x7f06c00d3570 2024-03-18T14:42:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.215+0000 7f06bdd9b640 1 -- 172.21.15.6:0/2426858769 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4031be0 con 0x7f06c00d3570 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 -- 172.21.15.6:0/2426858769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d3570 msgr2=0x7f06c00b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 --2- 172.21.15.6:0/2426858769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d3570 0x7f06c00b6140 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f06b40097a0 tx=0x7f06b402b8b0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 -- 172.21.15.6:0/2426858769 shutdown_connections 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 --2- 172.21.15.6:0/2426858769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c00d0450 0x7f06c00d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 --2- 172.21.15.6:0/2426858769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d3570 0x7f06c00b6140 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 -- 172.21.15.6:0/2426858769 >> 172.21.15.6:0/2426858769 conn(0x7f06c006c8c0 msgr2=0x7f06c006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 -- 172.21.15.6:0/2426858769 shutdown_connections 2024-03-18T14:42:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.216+0000 7f06c5259640 1 -- 172.21.15.6:0/2426858769 wait complete. 2024-03-18T14:42:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.217+0000 7f06c5259640 1 Processor -- start 2024-03-18T14:42:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.217+0000 7f06c5259640 1 -- start start 2024-03-18T14:42:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06c5259640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d0450 0x7f06c015af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06c5259640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06c00d3570 0x7f06c00cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06c5259640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c015e030 0x7f06c014fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06c5259640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f06c00ce550 con 0x7f06c00d0450 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06c5259640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f06c00d3290 con 0x7f06c00d3570 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06c5259640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f06c00ce830 con 0x7f06c015e030 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06bed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d0450 0x7f06c015af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06bed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d0450 0x7f06c015af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36954/0 (socket says 172.21.15.6:36954) 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06bed9d640 1 -- 172.21.15.6:0/3876789423 learned_addr learned my addr 172.21.15.6:0/3876789423 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06be59c640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06c00d3570 0x7f06c00cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.218+0000 7f06bf59e640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c015e030 0x7f06c014fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06bed9d640 1 -- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c015e030 msgr2=0x7f06c014fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06bed9d640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c015e030 0x7f06c014fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06bed9d640 1 -- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06c00d3570 msgr2=0x7f06c00cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06bed9d640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06c00d3570 0x7f06c00cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06bed9d640 1 -- 172.21.15.6:0/3876789423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c00cc240 con 0x7f06c00d0450 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06bed9d640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d0450 0x7f06c015af30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f06b800c430 tx=0x7f06b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06a77fe640 1 -- 172.21.15.6:0/3876789423 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b80074e0 con 0x7f06c00d0450 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06a77fe640 1 -- 172.21.15.6:0/3876789423 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b80047e0 con 0x7f06c00d0450 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c00cc520 con 0x7f06c00d0450 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06a77fe640 1 -- 172.21.15.6:0/3876789423 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8004bd0 con 0x7f06c00d0450 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.219+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c00ccae0 con 0x7f06c00d0450 2024-03-18T14:42:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.220+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c00751b0 con 0x7f06c00d0450 2024-03-18T14:42:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.221+0000 7f06a77fe640 1 -- 172.21.15.6:0/3876789423 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f06b8011070 con 0x7f06c00d0450 2024-03-18T14:42:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.221+0000 7f06a77fe640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f06ac06bbd0 0x7f06ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.221+0000 7f06a77fe640 1 -- 172.21.15.6:0/3876789423 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f06b808a8f0 con 0x7f06c00d0450 2024-03-18T14:42:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.221+0000 7f06be59c640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f06ac06bbd0 0x7f06ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.222+0000 7f06be59c640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f06ac06bbd0 0x7f06ac06e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f06b4036000 tx=0x7f06b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.225+0000 7f06a77fe640 1 -- 172.21.15.6:0/3876789423 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f06b8059070 con 0x7f06c00d0450 2024-03-18T14:42:29.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.408+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f06c00d19f0 con 0x7f06c00d0450 2024-03-18T14:42:29.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.409+0000 7f06a77fe640 1 -- 172.21.15.6:0/3876789423 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f06b8058970 con 0x7f06c00d0450 2024-03-18T14:42:29.412 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f06ac06bbd0 msgr2=0x7f06ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f06ac06bbd0 0x7f06ac06e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f06b4036000 tx=0x7f06b4008030 comp rx=0 tx=0).stop 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d0450 msgr2=0x7f06c015af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d0450 0x7f06c015af30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f06b800c430 tx=0x7f06b800c830 comp rx=0 tx=0).stop 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 shutdown_connections 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f06ac06bbd0 0x7f06ac06e090 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06c015e030 0x7f06c014fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06c00d3570 0x7f06c00cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 --2- 172.21.15.6:0/3876789423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06c00d0450 0x7f06c015af30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 >> 172.21.15.6:0/3876789423 conn(0x7f06c006c8c0 msgr2=0x7f06c006cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 shutdown_connections 2024-03-18T14:42:29.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.411+0000 7f06c5259640 1 -- 172.21.15.6:0/3876789423 wait complete. 2024-03-18T14:42:29.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:29.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.526+0000 7ff1d4a7e640 1 Processor -- start 2024-03-18T14:42:29.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.526+0000 7ff1d4a7e640 1 -- start start 2024-03-18T14:42:29.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1d4a7e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff1d0161740 0x7ff1d009ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1d4a7e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d00b4920 con 0x7ff1d006cef0 2024-03-18T14:42:29.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1d4a7e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d006d740 con 0x7ff1d009a3c0 2024-03-18T14:42:29.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1d4a7e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff1d00cc120 con 0x7ff1d0161740 2024-03-18T14:42:29.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1cdd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff1d006cef0 0x7ff1d0099cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55990/0 (socket says 172.21.15.6:55990) 2024-03-18T14:42:29.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1cdd9b640 1 -- 172.21.15.6:0/2861324540 learned_addr learned my addr 172.21.15.6:0/2861324540 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:29.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1ce59c640 1 --2- 172.21.15.6:0/2861324540 >> v2:172.21.15.92:3300/0 conn(0x7ff1d0161740 0x7ff1d009ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1cd59a640 1 -- 172.21.15.6:0/2861324540 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912882201 0 0) 0x7ff1b8001110 con 0x7ff1d006cef0 2024-03-18T14:42:29.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1cd59a640 1 -- 172.21.15.6:0/2861324540 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc003520 con 0x7ff1d006cef0 2024-03-18T14:42:29.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1ce59c640 1 -- 172.21.15.6:0/2861324540 >> v1:172.21.15.92:6789/0 conn(0x7ff1d009a3c0 legacy=0x7ff1d00cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1cd59a640 1 -- 172.21.15.6:0/2861324540 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2314552056 0 0) 0x7ff1d006d740 con 0x7ff1d009a3c0 2024-03-18T14:42:29.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1ce59c640 1 -- 172.21.15.6:0/2861324540 >> v1:172.21.15.6:6789/0 conn(0x7ff1d006cef0 legacy=0x7ff1d0099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.527+0000 7ff1ce59c640 1 -- 172.21.15.6:0/2861324540 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d00cc370 con 0x7ff1d0161740 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.528+0000 7ff1ce59c640 1 --2- 172.21.15.6:0/2861324540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0161740 0x7ff1d009ad30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097a0 tx=0x7ff1c802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c32409d5d98ab32 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.528+0000 7ff1cd59a640 1 -- 172.21.15.6:0/2861324540 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8034070 con 0x7ff1d0161740 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.528+0000 7ff1cd59a640 1 -- 172.21.15.6:0/2861324540 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c8030610 con 0x7ff1d0161740 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.528+0000 7ff1cd59a640 1 -- 172.21.15.6:0/2861324540 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80309c0 con 0x7ff1d0161740 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.529+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/2861324540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0161740 msgr2=0x7ff1d009ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.529+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/2861324540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0161740 0x7ff1d009ad30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097a0 tx=0x7ff1c802b960 comp rx=0 tx=0).stop 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.529+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/2861324540 shutdown_connections 2024-03-18T14:42:29.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.529+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/2861324540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0161740 0x7ff1d009ad30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.529+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/2861324540 >> 172.21.15.6:0/2861324540 conn(0x7ff1d00b35c0 msgr2=0x7ff1d00b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:29.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.529+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/2861324540 shutdown_connections 2024-03-18T14:42:29.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.529+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/2861324540 wait complete. 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.530+0000 7ff1d4a7e640 1 Processor -- start 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.530+0000 7ff1d4a7e640 1 -- start start 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1d4a7e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1d006cef0 0x7ff1d0155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1d4a7e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d009a3c0 0x7ff1d00c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1d4a7e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1d0161740 0x7ff1d00c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1d4a7e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d006d330 con 0x7ff1d006cef0 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1d4a7e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d00cc930 con 0x7ff1d009a3c0 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1d4a7e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d006d4b0 con 0x7ff1d0161740 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1ce59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1d006cef0 0x7ff1d0155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1ce59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1d006cef0 0x7ff1d0155090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36966/0 (socket says 172.21.15.6:36966) 2024-03-18T14:42:29.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1ce59c640 1 -- 172.21.15.6:0/1325374094 learned_addr learned my addr 172.21.15.6:0/1325374094 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1cdd9b640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d009a3c0 0x7ff1d00c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1ced9d640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1d0161740 0x7ff1d00c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1ce59c640 1 -- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1d0161740 msgr2=0x7ff1d00c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.531+0000 7ff1ce59c640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1d0161740 0x7ff1d00c5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1ce59c640 1 -- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d009a3c0 msgr2=0x7ff1d00c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1ce59c640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d009a3c0 0x7ff1d00c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1ce59c640 1 -- 172.21.15.6:0/1325374094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d00c7e40 con 0x7ff1d006cef0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1ce59c640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1d006cef0 0x7ff1d0155090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8000c00 tx=0x7ff1c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1c6ffd640 1 -- 172.21.15.6:0/1325374094 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8034040 con 0x7ff1d006cef0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1c6ffd640 1 -- 172.21.15.6:0/1325374094 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c8030410 con 0x7ff1d006cef0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d00c8120 con 0x7ff1d006cef0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1c6ffd640 1 -- 172.21.15.6:0/1325374094 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8039c30 con 0x7ff1d006cef0 2024-03-18T14:42:29.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.532+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d0157060 con 0x7ff1d006cef0 2024-03-18T14:42:29.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.533+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d00751b0 con 0x7ff1d006cef0 2024-03-18T14:42:29.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.533+0000 7ff1c6ffd640 1 -- 172.21.15.6:0/1325374094 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff1c8037070 con 0x7ff1d006cef0 2024-03-18T14:42:29.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.534+0000 7ff1c6ffd640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff1b006bbd0 0x7ff1b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.534+0000 7ff1c6ffd640 1 -- 172.21.15.6:0/1325374094 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff1c80accc0 con 0x7ff1d006cef0 2024-03-18T14:42:29.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.534+0000 7ff1cdd9b640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff1b006bbd0 0x7ff1b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.534+0000 7ff1cdd9b640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff1b006bbd0 0x7ff1b006e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00b41a0 tx=0x7ff1b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.538+0000 7ff1c6ffd640 1 -- 172.21.15.6:0/1325374094 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff1c807ae80 con 0x7ff1d006cef0 2024-03-18T14:42:29.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.721+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff1d0071550 con 0x7ff1d006cef0 2024-03-18T14:42:29.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.721+0000 7ff1c6ffd640 1 -- 172.21.15.6:0/1325374094 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7ff1c807ef20 con 0x7ff1d006cef0 2024-03-18T14:42:29.724 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff1b006bbd0 msgr2=0x7ff1b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff1b006bbd0 0x7ff1b006e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00b41a0 tx=0x7ff1b8009040 comp rx=0 tx=0).stop 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1d006cef0 msgr2=0x7ff1d0155090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1d006cef0 0x7ff1d0155090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8000c00 tx=0x7ff1c80046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 shutdown_connections 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff1b006bbd0 0x7ff1b006e090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1d0161740 0x7ff1d00c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d009a3c0 0x7ff1d00c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 --2- 172.21.15.6:0/1325374094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1d006cef0 0x7ff1d0155090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 >> 172.21.15.6:0/1325374094 conn(0x7ff1d00b35c0 msgr2=0x7ff1d0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:29.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 shutdown_connections 2024-03-18T14:42:29.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.723+0000 7ff1d4a7e640 1 -- 172.21.15.6:0/1325374094 wait complete. 2024-03-18T14:42:29.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:29.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.837+0000 7f12a5ec6640 1 Processor -- start 2024-03-18T14:42:29.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.837+0000 7f12a5ec6640 1 -- start start 2024-03-18T14:42:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f12a5ec6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f12a00cecc0 0x7f12a015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f12a5ec6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a00d12a0 con 0x7f12a015dd70 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f12a5ec6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a0161a50 con 0x7f12a00d0ae0 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f12a5ec6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f12a0161d30 con 0x7f12a00cecc0 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12a015dd70 0x7f12a00d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56006/0 (socket says 172.21.15.6:56006) 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129f7fe640 1 -- 172.21.15.6:0/3229342773 learned_addr learned my addr 172.21.15.6:0/3229342773 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129effd640 1 --2- 172.21.15.6:0/3229342773 >> v2:172.21.15.92:3300/0 conn(0x7f12a00cecc0 0x7f12a015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129e7fc640 1 -- 172.21.15.6:0/3229342773 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3448181550 0 0) 0x7f12a00d12a0 con 0x7f12a015dd70 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129e7fc640 1 -- 172.21.15.6:0/3229342773 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1288003520 con 0x7f12a015dd70 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129e7fc640 1 -- 172.21.15.6:0/3229342773 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3822351211 0 0) 0x7f12a0161a50 con 0x7f12a00d0ae0 2024-03-18T14:42:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129e7fc640 1 -- 172.21.15.6:0/3229342773 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a00d12a0 con 0x7f12a00d0ae0 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129effd640 1 -- 172.21.15.6:0/3229342773 >> v1:172.21.15.92:6789/0 conn(0x7f12a00d0ae0 legacy=0x7f12a00b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129effd640 1 -- 172.21.15.6:0/3229342773 >> v1:172.21.15.6:6789/0 conn(0x7f12a015dd70 legacy=0x7f12a00d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.838+0000 7f129effd640 1 -- 172.21.15.6:0/3229342773 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a0162010 con 0x7f12a00cecc0 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.839+0000 7f129effd640 1 --2- 172.21.15.6:0/3229342773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a00cecc0 0x7f12a015ee60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12940097a0 tx=0x7f129402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4862991d6108091 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.839+0000 7f129e7fc640 1 -- 172.21.15.6:0/3229342773 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294034070 con 0x7f12a00cecc0 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.839+0000 7f129e7fc640 1 -- 172.21.15.6:0/3229342773 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1294030610 con 0x7f12a00cecc0 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.839+0000 7f129e7fc640 1 -- 172.21.15.6:0/3229342773 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12940309c0 con 0x7f12a00cecc0 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.840+0000 7f12a5ec6640 1 -- 172.21.15.6:0/3229342773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a00cecc0 msgr2=0x7f12a015ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.840+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/3229342773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a00cecc0 0x7f12a015ee60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12940097a0 tx=0x7f129402b960 comp rx=0 tx=0).stop 2024-03-18T14:42:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.840+0000 7f12a5ec6640 1 -- 172.21.15.6:0/3229342773 shutdown_connections 2024-03-18T14:42:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.840+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/3229342773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a00cecc0 0x7f12a015ee60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.840+0000 7f12a5ec6640 1 -- 172.21.15.6:0/3229342773 >> 172.21.15.6:0/3229342773 conn(0x7f12a006c900 msgr2=0x7f12a006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.840+0000 7f12a5ec6640 1 -- 172.21.15.6:0/3229342773 shutdown_connections 2024-03-18T14:42:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.840+0000 7f12a5ec6640 1 -- 172.21.15.6:0/3229342773 wait complete. 2024-03-18T14:42:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.841+0000 7f12a5ec6640 1 Processor -- start 2024-03-18T14:42:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.841+0000 7f12a5ec6640 1 -- start start 2024-03-18T14:42:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f12a5ec6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12a00cecc0 0x7f12a01f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f12a5ec6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12a00d0ae0 0x7f12a014b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f12a5ec6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a015dd70 0x7f12a00c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f12a5ec6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f12a015e2d0 con 0x7f12a00cecc0 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f12a5ec6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f12a015e150 con 0x7f12a015dd70 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f12a5ec6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f12a00d2e10 con 0x7f12a00d0ae0 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12a00cecc0 0x7f12a01f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12a00cecc0 0x7f12a01f38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36972/0 (socket says 172.21.15.6:36972) 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129ffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a015dd70 0x7f12a00c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 -- 172.21.15.6:0/390047016 learned_addr learned my addr 172.21.15.6:0/390047016 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129effd640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12a00d0ae0 0x7f12a014b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 -- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12a00d0ae0 msgr2=0x7f12a014b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12a00d0ae0 0x7f12a014b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 -- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a015dd70 msgr2=0x7f12a00c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a015dd70 0x7f12a00c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 -- 172.21.15.6:0/390047016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a01f4840 con 0x7f12a00cecc0 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129ffff640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a015dd70 0x7f12a00c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.842+0000 7f129f7fe640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12a00cecc0 0x7f12a01f38e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f129000c430 tx=0x7f129000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.843+0000 7f1287fff640 1 -- 172.21.15.6:0/390047016 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290017070 con 0x7f12a00cecc0 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.843+0000 7f1287fff640 1 -- 172.21.15.6:0/390047016 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1290005d00 con 0x7f12a00cecc0 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.843+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a01f3fa0 con 0x7f12a00cecc0 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.843+0000 7f1287fff640 1 -- 172.21.15.6:0/390047016 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290007310 con 0x7f12a00cecc0 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.843+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a01f4560 con 0x7f12a00cecc0 2024-03-18T14:42:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.844+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a004f5b0 con 0x7f12a00cecc0 2024-03-18T14:42:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.844+0000 7f1287fff640 1 -- 172.21.15.6:0/390047016 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f129001a070 con 0x7f12a00cecc0 2024-03-18T14:42:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.844+0000 7f1287fff640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f128c06bbd0 0x7f128c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.845+0000 7f1287fff640 1 -- 172.21.15.6:0/390047016 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f129008a290 con 0x7f12a00cecc0 2024-03-18T14:42:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.845+0000 7f129effd640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f128c06bbd0 0x7f128c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.845+0000 7f129effd640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f128c06bbd0 0x7f128c06e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1294036000 tx=0x7f129403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:29.849+0000 7f1287fff640 1 -- 172.21.15.6:0/390047016 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f12900582b0 con 0x7f12a00cecc0 2024-03-18T14:42:30.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.042+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f12a015e580 con 0x7f12a00cecc0 2024-03-18T14:42:30.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.042+0000 7f1287fff640 1 -- 172.21.15.6:0/390047016 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f129005c350 con 0x7f12a00cecc0 2024-03-18T14:42:30.045 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f128c06bbd0 msgr2=0x7f128c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f128c06bbd0 0x7f128c06e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1294036000 tx=0x7f129403b040 comp rx=0 tx=0).stop 2024-03-18T14:42:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12a00cecc0 msgr2=0x7f12a01f38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12a00cecc0 0x7f12a01f38e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f129000c430 tx=0x7f129000c8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 shutdown_connections 2024-03-18T14:42:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f128c06bbd0 0x7f128c06e090 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a015dd70 0x7f12a00c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12a00d0ae0 0x7f12a014b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 --2- 172.21.15.6:0/390047016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12a00cecc0 0x7f12a01f38e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 >> 172.21.15.6:0/390047016 conn(0x7f12a006c900 msgr2=0x7f12a015ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:30.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 shutdown_connections 2024-03-18T14:42:30.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.044+0000 7f12a5ec6640 1 -- 172.21.15.6:0/390047016 wait complete. 2024-03-18T14:42:30.058 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:42:30.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:30.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T14:42:30.084 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:30.084 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:42:30.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:30.085 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:42:30.139 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:42:30.139 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:30.155 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:42:30.155 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:30.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:30.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.310+0000 7fc2e08ea640 1 Processor -- start 2024-03-18T14:42:30.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.310+0000 7fc2e08ea640 1 -- start start 2024-03-18T14:42:30.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2e08ea640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2dc06b8d0 0x7fc2dc0cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2e08ea640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc0afd00 0x7fc2dc0cd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2e08ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2dc0cce30 con 0x7fc2dc15e240 2024-03-18T14:42:30.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2e08ea640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc0cb000 con 0x7fc2dc06b8d0 2024-03-18T14:42:30.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2e08ea640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b1f90 con 0x7fc2dc0afd00 2024-03-18T14:42:30.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2d9d9b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2dc06b8d0 0x7fc2dc0cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2da59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc2dc15e240 0x7fc2dc06b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56020/0 (socket says 172.21.15.6:56020) 2024-03-18T14:42:30.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2da59c640 1 -- 172.21.15.6:0/3029503909 learned_addr learned my addr 172.21.15.6:0/3029503909 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:30.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2dad9d640 1 --2- 172.21.15.6:0/3029503909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc0afd00 0x7fc2dc0cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2d9d9b640 1 -- 172.21.15.6:0/3029503909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc0afd00 msgr2=0x7fc2dc0cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:30.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2d9d9b640 1 --2- 172.21.15.6:0/3029503909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc0afd00 0x7fc2dc0cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2d9d9b640 1 -- 172.21.15.6:0/3029503909 >> v1:172.21.15.6:6789/0 conn(0x7fc2dc15e240 legacy=0x7fc2dc06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2d9d9b640 1 -- 172.21.15.6:0/3029503909 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc0b1c20 con 0x7fc2dc06b8d0 2024-03-18T14:42:30.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.311+0000 7fc2d959a640 1 -- 172.21.15.6:0/3029503909 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2358392850 0 0) 0x7fc2dc0cce30 con 0x7fc2dc15e240 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.312+0000 7fc2d9d9b640 1 --2- 172.21.15.6:0/3029503909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc06b8d0 0x7fc2dc0cd0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0b2110 tx=0x7fc2d002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17e5e95b8f13f17 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.312+0000 7fc2d959a640 1 -- 172.21.15.6:0/3029503909 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0031770 con 0x7fc2dc06b8d0 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.312+0000 7fc2d959a640 1 -- 172.21.15.6:0/3029503909 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d0037070 con 0x7fc2dc06b8d0 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.312+0000 7fc2d959a640 1 -- 172.21.15.6:0/3029503909 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0031b30 con 0x7fc2dc06b8d0 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 -- 172.21.15.6:0/3029503909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc06b8d0 msgr2=0x7fc2dc0cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/3029503909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc06b8d0 0x7fc2dc0cd0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0b2110 tx=0x7fc2d002b810 comp rx=0 tx=0).stop 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 -- 172.21.15.6:0/3029503909 shutdown_connections 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/3029503909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc0afd00 0x7fc2dc0cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/3029503909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc06b8d0 0x7fc2dc0cd0d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 -- 172.21.15.6:0/3029503909 >> 172.21.15.6:0/3029503909 conn(0x7fc2dc0d09b0 msgr2=0x7fc2dc0d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 -- 172.21.15.6:0/3029503909 shutdown_connections 2024-03-18T14:42:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.313+0000 7fc2e08ea640 1 -- 172.21.15.6:0/3029503909 wait complete. 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.314+0000 7fc2e08ea640 1 Processor -- start 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.314+0000 7fc2e08ea640 1 -- start start 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2e08ea640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2dc06b8d0 0x7fc2dc154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2e08ea640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc0afd00 0x7fc2dc0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2e08ea640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc15e240 0x7fc2dc06fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2e08ea640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0ce580 con 0x7fc2dc0afd00 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2e08ea640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b1180 con 0x7fc2dc06b8d0 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2e08ea640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc0ce700 con 0x7fc2dc15e240 2024-03-18T14:42:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc0afd00 0x7fc2dc0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc0afd00 0x7fc2dc0ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36986/0 (socket says 172.21.15.6:36986) 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 -- 172.21.15.6:0/1588809288 learned_addr learned my addr 172.21.15.6:0/1588809288 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2dad9d640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc15e240 0x7fc2dc06fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2da59c640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2dc06b8d0 0x7fc2dc154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 -- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc15e240 msgr2=0x7fc2dc06fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc15e240 0x7fc2dc06fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 -- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2dc06b8d0 msgr2=0x7fc2dc154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2dc06b8d0 0x7fc2dc154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.315+0000 7fc2d9d9b640 1 -- 172.21.15.6:0/1588809288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc15eb90 con 0x7fc2dc0afd00 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.316+0000 7fc2d9d9b640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc0afd00 0x7fc2dc0ca220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009890 tx=0x7fc2d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.316+0000 7fc2caffd640 1 -- 172.21.15.6:0/1588809288 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0032600 con 0x7fc2dc0afd00 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.316+0000 7fc2caffd640 1 -- 172.21.15.6:0/1588809288 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d00327c0 con 0x7fc2dc0afd00 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.316+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2dc15ee70 con 0x7fc2dc0afd00 2024-03-18T14:42:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.316+0000 7fc2caffd640 1 -- 172.21.15.6:0/1588809288 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0032bb0 con 0x7fc2dc0afd00 2024-03-18T14:42:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.316+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc0c83f0 con 0x7fc2dc0afd00 2024-03-18T14:42:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.317+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc09e9a0 con 0x7fc2dc0afd00 2024-03-18T14:42:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.317+0000 7fc2caffd640 1 -- 172.21.15.6:0/1588809288 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc2d0031070 con 0x7fc2dc0afd00 2024-03-18T14:42:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.317+0000 7fc2caffd640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2bc06bbd0 0x7fc2bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.317+0000 7fc2caffd640 1 -- 172.21.15.6:0/1588809288 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc2d00ac6e0 con 0x7fc2dc0afd00 2024-03-18T14:42:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.318+0000 7fc2da59c640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2bc06bbd0 0x7fc2bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.318+0000 7fc2da59c640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2bc06bbd0 0x7fc2bc06e090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4007fd0 tx=0x7fc2d4007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.322+0000 7fc2caffd640 1 -- 172.21.15.6:0/1588809288 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc2d0040070 con 0x7fc2dc0afd00 2024-03-18T14:42:30.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.505+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2dc04f620 con 0x7fc2dc0afd00 2024-03-18T14:42:30.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.506+0000 7fc2caffd640 1 -- 172.21.15.6:0/1588809288 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fc2d007a770 con 0x7fc2dc0afd00 2024-03-18T14:42:30.509 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:30.509 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2bc06bbd0 msgr2=0x7fc2bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2bc06bbd0 0x7fc2bc06e090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4007fd0 tx=0x7fc2d4007de0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc0afd00 msgr2=0x7fc2dc0ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc0afd00 0x7fc2dc0ca220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009890 tx=0x7fc2d00046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 shutdown_connections 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc2bc06bbd0 0x7fc2bc06e090 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2dc15e240 0x7fc2dc06fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2dc0afd00 0x7fc2dc0ca220 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 --2- 172.21.15.6:0/1588809288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2dc06b8d0 0x7fc2dc154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 >> 172.21.15.6:0/1588809288 conn(0x7fc2dc0d09b0 msgr2=0x7fc2dc0b0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 shutdown_connections 2024-03-18T14:42:30.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.508+0000 7fc2e08ea640 1 -- 172.21.15.6:0/1588809288 wait complete. 2024-03-18T14:42:30.521 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:30.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:30.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.624+0000 7f05410dd640 1 Processor -- start 2024-03-18T14:42:30.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.624+0000 7f05410dd640 1 -- start start 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f05410dd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f053c0b0ea0 0x7f053c165af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f05410dd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c15f6c0 con 0x7f053c143ed0 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f05410dd640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c15e900 con 0x7f053c1443c0 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f05410dd640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f053c0cf7d0 con 0x7f053c0b0ea0 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f053b59e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f053c0b0ea0 0x7f053c165af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f053ad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f053c143ed0 0x7f053c0cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56028/0 (socket says 172.21.15.6:56028) 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f053ad9d640 1 -- 172.21.15.6:0/3650770348 learned_addr learned my addr 172.21.15.6:0/3650770348 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f053b59e640 1 -- 172.21.15.6:0/3650770348 >> v1:172.21.15.92:6789/0 conn(0x7f053c1443c0 legacy=0x7f053c0b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f053b59e640 1 -- 172.21.15.6:0/3650770348 >> v1:172.21.15.6:6789/0 conn(0x7f053c143ed0 legacy=0x7f053c0cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f0539d9b640 1 -- 172.21.15.6:0/3650770348 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658915691 0 0) 0x7f053c15e900 con 0x7f053c1443c0 2024-03-18T14:42:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.625+0000 7f053b59e640 1 -- 172.21.15.6:0/3650770348 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c15ebe0 con 0x7f053c0b0ea0 2024-03-18T14:42:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.626+0000 7f053b59e640 1 --2- 172.21.15.6:0/3650770348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c0b0ea0 0x7f053c165af0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f052c009d90 tx=0x7f052c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9d367be2548a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.626+0000 7f0539d9b640 1 -- 172.21.15.6:0/3650770348 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c039070 con 0x7f053c0b0ea0 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.626+0000 7f0539d9b640 1 -- 172.21.15.6:0/3650770348 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f052c02cab0 con 0x7f053c0b0ea0 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.626+0000 7f0539d9b640 1 -- 172.21.15.6:0/3650770348 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c034440 con 0x7f053c0b0ea0 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.626+0000 7f05410dd640 1 -- 172.21.15.6:0/3650770348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c0b0ea0 msgr2=0x7f053c165af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.626+0000 7f05410dd640 1 --2- 172.21.15.6:0/3650770348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c0b0ea0 0x7f053c165af0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f052c009d90 tx=0x7f052c02bd80 comp rx=0 tx=0).stop 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.627+0000 7f05410dd640 1 -- 172.21.15.6:0/3650770348 shutdown_connections 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.627+0000 7f05410dd640 1 --2- 172.21.15.6:0/3650770348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c0b0ea0 0x7f053c165af0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.627+0000 7f05410dd640 1 -- 172.21.15.6:0/3650770348 >> 172.21.15.6:0/3650770348 conn(0x7f053c06b1c0 msgr2=0x7f053c06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.627+0000 7f05410dd640 1 -- 172.21.15.6:0/3650770348 shutdown_connections 2024-03-18T14:42:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.627+0000 7f05410dd640 1 -- 172.21.15.6:0/3650770348 wait complete. 2024-03-18T14:42:30.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.628+0000 7f05410dd640 1 Processor -- start 2024-03-18T14:42:30.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.628+0000 7f05410dd640 1 -- start start 2024-03-18T14:42:30.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f05410dd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f053c0b0ea0 0x7f053c13c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f05410dd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f053c143ed0 0x7f053c15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f05410dd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c1443c0 0x7f053c0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f05410dd640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f053c15f230 con 0x7f053c1443c0 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f05410dd640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f053c0cf590 con 0x7f053c143ed0 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f05410dd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f053c0b4880 con 0x7f053c0b0ea0 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c1443c0 0x7f053c0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053ad9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f053c0b0ea0 0x7f053c13c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c1443c0 0x7f053c0afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36996/0 (socket says 172.21.15.6:36996) 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 -- 172.21.15.6:0/1179833832 learned_addr learned my addr 172.21.15.6:0/1179833832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053a59c640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f053c143ed0 0x7f053c15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 -- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f053c0b0ea0 msgr2=0x7f053c13c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f053c0b0ea0 0x7f053c13c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 -- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f053c143ed0 msgr2=0x7f053c15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f053c143ed0 0x7f053c15c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 -- 172.21.15.6:0/1179833832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c15c730 con 0x7f053c1443c0 2024-03-18T14:42:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.629+0000 7f053b59e640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c1443c0 0x7f053c0afdf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f052c009d90 tx=0x7f052c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.630+0000 7f05237fe640 1 -- 172.21.15.6:0/1179833832 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c044030 con 0x7f053c1443c0 2024-03-18T14:42:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.630+0000 7f05237fe640 1 -- 172.21.15.6:0/1179833832 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f052c02c8b0 con 0x7f053c1443c0 2024-03-18T14:42:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.630+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053c15ca10 con 0x7f053c1443c0 2024-03-18T14:42:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.630+0000 7f05237fe640 1 -- 172.21.15.6:0/1179833832 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c039040 con 0x7f053c1443c0 2024-03-18T14:42:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.630+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053c168cc0 con 0x7f053c1443c0 2024-03-18T14:42:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.630+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053c15cbe0 con 0x7f053c1443c0 2024-03-18T14:42:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.631+0000 7f05237fe640 1 -- 172.21.15.6:0/1179833832 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f052c047070 con 0x7f053c1443c0 2024-03-18T14:42:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.631+0000 7f05237fe640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f052806bbd0 0x7f052806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.631+0000 7f05237fe640 1 -- 172.21.15.6:0/1179833832 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f052c0ad620 con 0x7f053c1443c0 2024-03-18T14:42:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.632+0000 7f053ad9d640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f052806bbd0 0x7f052806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.632+0000 7f053ad9d640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f052806bbd0 0x7f052806e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0534000f60 tx=0x7f0534007e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.636+0000 7f05237fe640 1 -- 172.21.15.6:0/1179833832 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f052c04e070 con 0x7f053c1443c0 2024-03-18T14:42:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.817+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f053c0751b0 con 0x7f053c1443c0 2024-03-18T14:42:30.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.817+0000 7f05237fe640 1 -- 172.21.15.6:0/1179833832 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f052c07b780 con 0x7f053c1443c0 2024-03-18T14:42:30.820 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f052806bbd0 msgr2=0x7f052806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f052806bbd0 0x7f052806e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0534000f60 tx=0x7f0534007e20 comp rx=0 tx=0).stop 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c1443c0 msgr2=0x7f053c0afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c1443c0 0x7f053c0afdf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f052c009d90 tx=0x7f052c004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 shutdown_connections 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f052806bbd0 0x7f052806e090 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f053c1443c0 0x7f053c0afdf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f053c143ed0 0x7f053c15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 --2- 172.21.15.6:0/1179833832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f053c0b0ea0 0x7f053c13c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 >> 172.21.15.6:0/1179833832 conn(0x7f053c06b1c0 msgr2=0x7f053c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:30.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 shutdown_connections 2024-03-18T14:42:30.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.819+0000 7f05410dd640 1 -- 172.21.15.6:0/1179833832 wait complete. 2024-03-18T14:42:30.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:30.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.936+0000 7f7bfba0b640 1 Processor -- start 2024-03-18T14:42:30.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.936+0000 7f7bfba0b640 1 -- start start 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bfba0b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7bf415df80 0x7f7bf40d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bfba0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40cea40 0x7f7bf40ceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bfba0b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bf40d2540 con 0x7f7bf40b98b0 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bfba0b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf40cf3f0 con 0x7f7bf415df80 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bfba0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf40d2820 con 0x7f7bf40cea40 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bf97a7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7bf415df80 0x7f7bf40d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bf8fa6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7bf40b98b0 0x7f7bf40d1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56042/0 (socket says 172.21.15.6:56042) 2024-03-18T14:42:30.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bf97a7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7bf415df80 0x7f7bf40d35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36998/0 (socket says 172.21.15.6:36998) 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bf8fa6640 1 -- 172.21.15.6:0/3094817328 learned_addr learned my addr 172.21.15.6:0/3094817328 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.937+0000 7f7bf9fa8640 1 --2- 172.21.15.6:0/3094817328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40cea40 0x7f7bf40ceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.938+0000 7f7bf97a7640 1 -- 172.21.15.6:0/3094817328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40cea40 msgr2=0x7f7bf40ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.938+0000 7f7bf97a7640 1 --2- 172.21.15.6:0/3094817328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40cea40 0x7f7bf40ceeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.938+0000 7f7bf97a7640 1 -- 172.21.15.6:0/3094817328 >> v1:172.21.15.6:6789/0 conn(0x7f7bf40b98b0 legacy=0x7f7bf40d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.938+0000 7f7bf3fff640 1 -- 172.21.15.6:0/3094817328 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1023319992 0 0) 0x7f7bf40d2540 con 0x7f7bf40b98b0 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.938+0000 7f7bf97a7640 1 -- 172.21.15.6:0/3094817328 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf40d2b00 con 0x7f7bf415df80 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.938+0000 7f7bf97a7640 1 --2- 172.21.15.6:0/3094817328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40d35e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7bec0097a0 tx=0x7f7bec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fd4a349a32b2fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.938+0000 7f7bf3fff640 1 -- 172.21.15.6:0/3094817328 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec035020 con 0x7f7bf415df80 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bf3fff640 1 -- 172.21.15.6:0/3094817328 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec031830 con 0x7f7bf415df80 2024-03-18T14:42:30.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bf3fff640 1 -- 172.21.15.6:0/3094817328 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec031c00 con 0x7f7bf415df80 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 -- 172.21.15.6:0/3094817328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 msgr2=0x7f7bf40d35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/3094817328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40d35e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7bec0097a0 tx=0x7f7bec02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 -- 172.21.15.6:0/3094817328 shutdown_connections 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/3094817328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40cea40 0x7f7bf40ceeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/3094817328 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40d35e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 -- 172.21.15.6:0/3094817328 >> 172.21.15.6:0/3094817328 conn(0x7f7bf4072d40 msgr2=0x7f7bf4073150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 -- 172.21.15.6:0/3094817328 shutdown_connections 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.939+0000 7f7bfba0b640 1 -- 172.21.15.6:0/3094817328 wait complete. 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.940+0000 7f7bfba0b640 1 Processor -- start 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.940+0000 7f7bfba0b640 1 -- start start 2024-03-18T14:42:30.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bfba0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40b98b0 0x7f7bf4161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bfba0b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bf40cea40 0x7f7bf40ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bfba0b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40acc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bfba0b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf415e360 con 0x7f7bf415df80 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bfba0b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf40d30c0 con 0x7f7bf40cea40 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bfba0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf40b6da0 con 0x7f7bf40b98b0 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40acc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37002/0 (socket says 172.21.15.6:37002) 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 -- 172.21.15.6:0/2509391934 learned_addr learned my addr 172.21.15.6:0/2509391934 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf8fa6640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bf40cea40 0x7f7bf40ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf97a7640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40b98b0 0x7f7bf4161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 -- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40b98b0 msgr2=0x7f7bf4161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40b98b0 0x7f7bf4161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 -- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bf40cea40 msgr2=0x7f7bf40ac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bf40cea40 0x7f7bf40ac6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.941+0000 7f7bf9fa8640 1 -- 172.21.15.6:0/2509391934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf415eef0 con 0x7f7bf415df80 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.942+0000 7f7bf9fa8640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40acc10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7be400efc0 tx=0x7f7be400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.942+0000 7f7bf1ffb640 1 -- 172.21.15.6:0/2509391934 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4018020 con 0x7f7bf415df80 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.942+0000 7f7bf1ffb640 1 -- 172.21.15.6:0/2509391934 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be4014480 con 0x7f7bf415df80 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.942+0000 7f7bf1ffb640 1 -- 172.21.15.6:0/2509391934 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4014870 con 0x7f7bf415df80 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.942+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf415f1d0 con 0x7f7bf415df80 2024-03-18T14:42:30.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.942+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf415f790 con 0x7f7bf415df80 2024-03-18T14:42:30.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.943+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf40751b0 con 0x7f7bf415df80 2024-03-18T14:42:30.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.943+0000 7f7bf1ffb640 1 -- 172.21.15.6:0/2509391934 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7be4005d10 con 0x7f7bf415df80 2024-03-18T14:42:30.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.944+0000 7f7bf1ffb640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7be006bbd0 0x7f7be006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:30.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.944+0000 7f7bf1ffb640 1 -- 172.21.15.6:0/2509391934 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7be408b8a0 con 0x7f7bf415df80 2024-03-18T14:42:30.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.944+0000 7f7bf97a7640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7be006bbd0 0x7f7be006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:30.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.944+0000 7f7bf97a7640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7be006bbd0 0x7f7be006e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bec036000 tx=0x7f7bec03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:30.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:30.948+0000 7f7bf1ffb640 1 -- 172.21.15.6:0/2509391934 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7be4022070 con 0x7f7bf415df80 2024-03-18T14:42:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.128+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7bf406b7f0 con 0x7f7bf415df80 2024-03-18T14:42:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.129+0000 7f7bf1ffb640 1 -- 172.21.15.6:0/2509391934 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7be4010040 con 0x7f7bf415df80 2024-03-18T14:42:31.131 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7be006bbd0 msgr2=0x7f7be006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7be006bbd0 0x7f7be006e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bec036000 tx=0x7f7bec03c040 comp rx=0 tx=0).stop 2024-03-18T14:42:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 msgr2=0x7f7bf40acc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40acc10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7be400efc0 tx=0x7f7be400c490 comp rx=0 tx=0).stop 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 shutdown_connections 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7be006bbd0 0x7f7be006e090 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf415df80 0x7f7bf40acc10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bf40cea40 0x7f7bf40ac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 --2- 172.21.15.6:0/2509391934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bf40b98b0 0x7f7bf4161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.130+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 >> 172.21.15.6:0/2509391934 conn(0x7f7bf4072d40 msgr2=0x7f7bf40b8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.131+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 shutdown_connections 2024-03-18T14:42:31.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.131+0000 7f7bfba0b640 1 -- 172.21.15.6:0/2509391934 wait complete. 2024-03-18T14:42:31.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:31.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.245+0000 7f6bd0537640 1 Processor -- start 2024-03-18T14:42:31.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.245+0000 7f6bd0537640 1 -- start start 2024-03-18T14:42:31.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bd0537640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc815dfd0 0x7f6bc80b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bd0537640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6bc80b7e00 0x7f6bc80b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bd0537640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bc815e3b0 con 0x7f6bc80b5740 2024-03-18T14:42:31.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bd0537640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6bc80b60e0 con 0x7f6bc80b7e00 2024-03-18T14:42:31.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bd0537640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc80b8470 con 0x7f6bc815dfd0 2024-03-18T14:42:31.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bce2d3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc815dfd0 0x7f6bc80b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bcead4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6bc80b7e00 0x7f6bc80b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bcead4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6bc80b7e00 0x7f6bc80b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46332/0 (socket says 172.21.15.6:46332) 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bce2d3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc815dfd0 0x7f6bc80b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51182/0 (socket says 172.21.15.6:51182) 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bcead4640 1 -- 172.21.15.6:0/4149738869 learned_addr learned my addr 172.21.15.6:0/4149738869 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bcdad2640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6bc80b5740 0x7f6bc80d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:50730/0 (socket says 172.21.15.6:50730) 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bcead4640 1 -- 172.21.15.6:0/4149738869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc815dfd0 msgr2=0x7f6bc80b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bcead4640 1 --2- 172.21.15.6:0/4149738869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc815dfd0 0x7f6bc80b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.246+0000 7f6bcead4640 1 -- 172.21.15.6:0/4149738869 >> v1:172.21.15.92:6789/0 conn(0x7f6bc80b5740 legacy=0x7f6bc80d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.247+0000 7f6bcead4640 1 -- 172.21.15.6:0/4149738869 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc80b8750 con 0x7f6bc80b7e00 2024-03-18T14:42:31.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.247+0000 7f6bcd2d1640 1 -- 172.21.15.6:0/4149738869 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507296070 0 0) 0x7f6bc815e3b0 con 0x7f6bc80b5740 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.247+0000 7f6bcead4640 1 --2- 172.21.15.6:0/4149738869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 0x7f6bc80b5ba0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009870 tx=0x7f6bc002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2febaf9574faf62b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.247+0000 7f6bcd2d1640 1 -- 172.21.15.6:0/4149738869 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0034070 con 0x7f6bc80b7e00 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.247+0000 7f6bcd2d1640 1 -- 172.21.15.6:0/4149738869 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bc0030840 con 0x7f6bc80b7e00 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bcd2d1640 1 -- 172.21.15.6:0/4149738869 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0030c10 con 0x7f6bc80b7e00 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 -- 172.21.15.6:0/4149738869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 msgr2=0x7f6bc80b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 --2- 172.21.15.6:0/4149738869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 0x7f6bc80b5ba0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009870 tx=0x7f6bc002b980 comp rx=0 tx=0).stop 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 -- 172.21.15.6:0/4149738869 shutdown_connections 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 --2- 172.21.15.6:0/4149738869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 0x7f6bc80b5ba0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 --2- 172.21.15.6:0/4149738869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc815dfd0 0x7f6bc80b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 -- 172.21.15.6:0/4149738869 >> 172.21.15.6:0/4149738869 conn(0x7f6bc8072d10 msgr2=0x7f6bc8073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 -- 172.21.15.6:0/4149738869 shutdown_connections 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.248+0000 7f6bd0537640 1 -- 172.21.15.6:0/4149738869 wait complete. 2024-03-18T14:42:31.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.249+0000 7f6bd0537640 1 Processor -- start 2024-03-18T14:42:31.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.249+0000 7f6bd0537640 1 -- start start 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bd0537640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc80b5740 0x7f6bc81f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bd0537640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 0x7f6bc81f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bd0537640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc815dfd0 0x7f6bc815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bd0537640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc80b9b90 con 0x7f6bc815dfd0 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bd0537640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc80b8d10 con 0x7f6bc80b7e00 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bd0537640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc80d2550 con 0x7f6bc80b5740 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcead4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc815dfd0 0x7f6bc815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcead4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc815dfd0 0x7f6bc815ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37010/0 (socket says 172.21.15.6:37010) 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcead4640 1 -- 172.21.15.6:0/2246939262 learned_addr learned my addr 172.21.15.6:0/2246939262 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bce2d3640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc80b5740 0x7f6bc81f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcdad2640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 0x7f6bc81f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcead4640 1 -- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc80b5740 msgr2=0x7f6bc81f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcead4640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc80b5740 0x7f6bc81f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcead4640 1 -- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 msgr2=0x7f6bc81f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.250+0000 7f6bcead4640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 0x7f6bc81f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bcead4640 1 -- 172.21.15.6:0/2246939262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc815fa00 con 0x7f6bc815dfd0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bce2d3640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc80b5740 0x7f6bc81f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bcead4640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc815dfd0 0x7f6bc815ef60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009870 tx=0x7f6bc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bc6ffd640 1 -- 172.21.15.6:0/2246939262 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0034040 con 0x7f6bc815dfd0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bc6ffd640 1 -- 172.21.15.6:0/2246939262 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bc0030640 con 0x7f6bc815dfd0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc80c77f0 con 0x7f6bc815dfd0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bc6ffd640 1 -- 172.21.15.6:0/2246939262 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0039cc0 con 0x7f6bc815dfd0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.251+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc80c8090 con 0x7f6bc815dfd0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.252+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc80c7be0 con 0x7f6bc815dfd0 2024-03-18T14:42:31.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.252+0000 7f6bc6ffd640 1 -- 172.21.15.6:0/2246939262 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6bc004b060 con 0x7f6bc815dfd0 2024-03-18T14:42:31.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.253+0000 7f6bc6ffd640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6bb406bbd0 0x7f6bb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.253+0000 7f6bc6ffd640 1 -- 172.21.15.6:0/2246939262 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6bc00ac920 con 0x7f6bc815dfd0 2024-03-18T14:42:31.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.253+0000 7f6bce2d3640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6bb406bbd0 0x7f6bb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.253+0000 7f6bce2d3640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6bb406bbd0 0x7f6bb406e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009fd0 tx=0x7f6bb800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.257+0000 7f6bc6ffd640 1 -- 172.21.15.6:0/2246939262 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6bc007aa50 con 0x7f6bc815dfd0 2024-03-18T14:42:31.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.438+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6bc80751b0 con 0x7f6bc815dfd0 2024-03-18T14:42:31.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.438+0000 7f6bc6ffd640 1 -- 172.21.15.6:0/2246939262 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f6bc007eaf0 con 0x7f6bc815dfd0 2024-03-18T14:42:31.441 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:31.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6bb406bbd0 msgr2=0x7f6bb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6bb406bbd0 0x7f6bb406e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009fd0 tx=0x7f6bb800b040 comp rx=0 tx=0).stop 2024-03-18T14:42:31.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc815dfd0 msgr2=0x7f6bc815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc815dfd0 0x7f6bc815ef60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009870 tx=0x7f6bc00046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 shutdown_connections 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6bb406bbd0 0x7f6bb406e090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc815dfd0 0x7f6bc815ef60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bc80b7e00 0x7f6bc81f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 --2- 172.21.15.6:0/2246939262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bc80b5740 0x7f6bc81f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 >> 172.21.15.6:0/2246939262 conn(0x7f6bc8072d10 msgr2=0x7f6bc80ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 shutdown_connections 2024-03-18T14:42:31.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.440+0000 7f6bd0537640 1 -- 172.21.15.6:0/2246939262 wait complete. 2024-03-18T14:42:31.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.556+0000 7f1b580cc640 1 Processor -- start 2024-03-18T14:42:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.556+0000 7f1b580cc640 1 -- start start 2024-03-18T14:42:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b580cc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b5015e030 0x7f1b500d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b580cc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b500b6140 0x7f1b500ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b580cc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b500d0450 0x7f1b500d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b580cc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1b5015e410 con 0x7f1b500d0450 2024-03-18T14:42:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b580cc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1b500ce300 con 0x7f1b500b6140 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b580cc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1b500d0e20 con 0x7f1b5015e030 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b56669640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b500d0450 0x7f1b500d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b55e68640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b5015e030 0x7f1b500d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b55667640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b500b6140 0x7f1b500ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b55e68640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b5015e030 0x7f1b500d3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51204/0 (socket says 172.21.15.6:51204) 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b55667640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b500b6140 0x7f1b500ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46346/0 (socket says 172.21.15.6:46346) 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b56669640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b500d0450 0x7f1b500d08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37026/0 (socket says 172.21.15.6:37026) 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.557+0000 7f1b55e68640 1 -- 172.21.15.6:0/2026863848 learned_addr learned my addr 172.21.15.6:0/2026863848 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b55e68640 1 -- 172.21.15.6:0/2026863848 >> v2:172.21.15.92:3300/0 conn(0x7f1b500b6140 msgr2=0x7f1b500ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b55e68640 1 --2- 172.21.15.6:0/2026863848 >> v2:172.21.15.92:3300/0 conn(0x7f1b500b6140 0x7f1b500ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b55e68640 1 -- 172.21.15.6:0/2026863848 >> v2:172.21.15.6:3300/0 conn(0x7f1b500d0450 msgr2=0x7f1b500d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b55e68640 1 --2- 172.21.15.6:0/2026863848 >> v2:172.21.15.6:3300/0 conn(0x7f1b500d0450 0x7f1b500d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b55e68640 1 -- 172.21.15.6:0/2026863848 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b500b65a0 con 0x7f1b5015e030 2024-03-18T14:42:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b56669640 1 --2- 172.21.15.6:0/2026863848 >> v2:172.21.15.6:3300/0 conn(0x7f1b500d0450 0x7f1b500d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b55e68640 1 --2- 172.21.15.6:0/2026863848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b5015e030 0x7f1b500d3570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1b5008c380 tx=0x7f1b4c02b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c25630b1c7d83fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.558+0000 7f1b54e66640 1 -- 172.21.15.6:0/2026863848 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c035020 con 0x7f1b5015e030 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b54e66640 1 -- 172.21.15.6:0/2026863848 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b4c02d040 con 0x7f1b5015e030 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b54e66640 1 -- 172.21.15.6:0/2026863848 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c031a80 con 0x7f1b5015e030 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 -- 172.21.15.6:0/2026863848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b5015e030 msgr2=0x7f1b500d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 --2- 172.21.15.6:0/2026863848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b5015e030 0x7f1b500d3570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1b5008c380 tx=0x7f1b4c02b900 comp rx=0 tx=0).stop 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 -- 172.21.15.6:0/2026863848 shutdown_connections 2024-03-18T14:42:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 --2- 172.21.15.6:0/2026863848 >> v2:172.21.15.6:3300/0 conn(0x7f1b500d0450 0x7f1b500d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 --2- 172.21.15.6:0/2026863848 >> v2:172.21.15.92:3300/0 conn(0x7f1b500b6140 0x7f1b500ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 --2- 172.21.15.6:0/2026863848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b5015e030 0x7f1b500d3570 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 -- 172.21.15.6:0/2026863848 >> 172.21.15.6:0/2026863848 conn(0x7f1b5006c8c0 msgr2=0x7f1b5006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 -- 172.21.15.6:0/2026863848 shutdown_connections 2024-03-18T14:42:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.559+0000 7f1b580cc640 1 -- 172.21.15.6:0/2026863848 wait complete. 2024-03-18T14:42:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.560+0000 7f1b580cc640 1 Processor -- start 2024-03-18T14:42:31.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.560+0000 7f1b580cc640 1 -- start start 2024-03-18T14:42:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b580cc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b500b6140 0x7f1b50156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b580cc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b500d0450 0x7f1b501f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b580cc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5015e030 0x7f1b500d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b580cc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1b500ce550 con 0x7f1b5015e030 2024-03-18T14:42:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b580cc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1b500d3290 con 0x7f1b500b6140 2024-03-18T14:42:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b580cc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1b500ce830 con 0x7f1b500d0450 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5015e030 0x7f1b500d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b55e68640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b500b6140 0x7f1b50156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5015e030 0x7f1b500d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37042/0 (socket says 172.21.15.6:37042) 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b55667640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b500d0450 0x7f1b501f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 -- 172.21.15.6:0/3011403889 learned_addr learned my addr 172.21.15.6:0/3011403889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 -- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b500d0450 msgr2=0x7f1b501f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b500d0450 0x7f1b501f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 -- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b500b6140 msgr2=0x7f1b50156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b500b6140 0x7f1b50156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.561+0000 7f1b56669640 1 -- 172.21.15.6:0/3011403889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b500b56b0 con 0x7f1b5015e030 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.562+0000 7f1b56669640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5015e030 0x7f1b500d1c10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800efc0 tx=0x7f1b48009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.562+0000 7f1b467fc640 1 -- 172.21.15.6:0/3011403889 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48018030 con 0x7f1b5015e030 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.562+0000 7f1b467fc640 1 -- 172.21.15.6:0/3011403889 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b4800cde0 con 0x7f1b5015e030 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.562+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b501f7110 con 0x7f1b5015e030 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.562+0000 7f1b467fc640 1 -- 172.21.15.6:0/3011403889 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48003bb0 con 0x7f1b5015e030 2024-03-18T14:42:31.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.562+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b501f76d0 con 0x7f1b5015e030 2024-03-18T14:42:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.563+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b500751b0 con 0x7f1b5015e030 2024-03-18T14:42:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.563+0000 7f1b467fc640 1 -- 172.21.15.6:0/3011403889 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1b48005d10 con 0x7f1b5015e030 2024-03-18T14:42:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.564+0000 7f1b467fc640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1b3406bbd0 0x7f1b3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.564+0000 7f1b467fc640 1 -- 172.21.15.6:0/3011403889 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1b4808b740 con 0x7f1b5015e030 2024-03-18T14:42:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.564+0000 7f1b55e68640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1b3406bbd0 0x7f1b3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.564+0000 7f1b55e68640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1b3406bbd0 0x7f1b3406e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c036000 tx=0x7f1b4c03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.568+0000 7f1b467fc640 1 -- 172.21.15.6:0/3011403889 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1b48022070 con 0x7f1b5015e030 2024-03-18T14:42:31.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.750+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1b500d19f0 con 0x7f1b5015e030 2024-03-18T14:42:31.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.751+0000 7f1b467fc640 1 -- 172.21.15.6:0/3011403889 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f1b48010040 con 0x7f1b5015e030 2024-03-18T14:42:31.753 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:31.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1b3406bbd0 msgr2=0x7f1b3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1b3406bbd0 0x7f1b3406e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c036000 tx=0x7f1b4c03c040 comp rx=0 tx=0).stop 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5015e030 msgr2=0x7f1b500d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5015e030 0x7f1b500d1c10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800efc0 tx=0x7f1b48009f90 comp rx=0 tx=0).stop 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 shutdown_connections 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1b3406bbd0 0x7f1b3406e090 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5015e030 0x7f1b500d1c10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b500d0450 0x7f1b501f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 --2- 172.21.15.6:0/3011403889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b500b6140 0x7f1b50156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 >> 172.21.15.6:0/3011403889 conn(0x7f1b5006c8c0 msgr2=0x7f1b500b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 shutdown_connections 2024-03-18T14:42:31.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.753+0000 7f1b580cc640 1 -- 172.21.15.6:0/3011403889 wait complete. 2024-03-18T14:42:31.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.866+0000 7f76e4b5b640 1 Processor -- start 2024-03-18T14:42:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.866+0000 7f76e4b5b640 1 -- start start 2024-03-18T14:42:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76e4b5b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76e015a440 0x7f76e00721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76e4b5b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e00b42a0 0x7f76e00cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76e4b5b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76e006da60 con 0x7f76e0072730 2024-03-18T14:42:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76e4b5b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f76e0072ba0 con 0x7f76e015a440 2024-03-18T14:42:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76e4b5b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f76e006dd40 con 0x7f76e00b42a0 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76de59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76e015a440 0x7f76e00721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76d7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f76e0072730 0x7f76e00b3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56050/0 (socket says 172.21.15.6:56050) 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76d7fff640 1 -- 172.21.15.6:0/3473982958 learned_addr learned my addr 172.21.15.6:0/3473982958 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76ded9d640 1 --2- 172.21.15.6:0/3473982958 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e00b42a0 0x7f76e00cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76de59c640 1 -- 172.21.15.6:0/3473982958 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e00b42a0 msgr2=0x7f76e00cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76de59c640 1 --2- 172.21.15.6:0/3473982958 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e00b42a0 0x7f76e00cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76de59c640 1 -- 172.21.15.6:0/3473982958 >> v1:172.21.15.6:6789/0 conn(0x7f76e0072730 legacy=0x7f76e00b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76ddd9b640 1 -- 172.21.15.6:0/3473982958 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2538220377 0 0) 0x7f76e006da60 con 0x7f76e0072730 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.867+0000 7f76de59c640 1 -- 172.21.15.6:0/3473982958 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e006e020 con 0x7f76e015a440 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.868+0000 7f76de59c640 1 --2- 172.21.15.6:0/3473982958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00721f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f76d00097a0 tx=0x7f76d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed050eb41a0aa4e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.868+0000 7f76ddd9b640 1 -- 172.21.15.6:0/3473982958 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0035020 con 0x7f76e015a440 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.868+0000 7f76ddd9b640 1 -- 172.21.15.6:0/3473982958 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d0031830 con 0x7f76e015a440 2024-03-18T14:42:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.868+0000 7f76ddd9b640 1 -- 172.21.15.6:0/3473982958 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0031c00 con 0x7f76e015a440 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3473982958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 msgr2=0x7f76e00721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3473982958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00721f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f76d00097a0 tx=0x7f76d002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3473982958 shutdown_connections 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3473982958 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e00b42a0 0x7f76e00cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3473982958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00721f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3473982958 >> 172.21.15.6:0/3473982958 conn(0x7f76e00b1570 msgr2=0x7f76e00b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3473982958 shutdown_connections 2024-03-18T14:42:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.869+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3473982958 wait complete. 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.870+0000 7f76e4b5b640 1 Processor -- start 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.870+0000 7f76e4b5b640 1 -- start start 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76e4b5b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e0072730 0x7f76e014b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76e4b5b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e00b42a0 0x7f76e00aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76e4b5b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76e4b5b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f76e015a820 con 0x7f76e015a440 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76e4b5b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f76e00b4810 con 0x7f76e00b42a0 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76e4b5b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f76e00b0c20 con 0x7f76e0072730 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00aaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37058/0 (socket says 172.21.15.6:37058) 2024-03-18T14:42:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 -- 172.21.15.6:0/3653234916 learned_addr learned my addr 172.21.15.6:0/3653234916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76d7fff640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e00b42a0 0x7f76e00aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76de59c640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e0072730 0x7f76e014b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 -- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e0072730 msgr2=0x7f76e014b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e0072730 0x7f76e014b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 -- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e00b42a0 msgr2=0x7f76e00aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e00b42a0 0x7f76e00aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 -- 172.21.15.6:0/3653234916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e00b0380 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76ded9d640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00aaa00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009dd0 tx=0x7f76d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.871+0000 7f76d6ffd640 1 -- 172.21.15.6:0/3653234916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d8018020 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.872+0000 7f76d6ffd640 1 -- 172.21.15.6:0/3653234916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d8014480 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.872+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e00b0660 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.872+0000 7f76d6ffd640 1 -- 172.21.15.6:0/3653234916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d8014870 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.872+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e014bbd0 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.873+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76e004f5b0 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.873+0000 7f76d6ffd640 1 -- 172.21.15.6:0/3653234916 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f76d8005d10 con 0x7f76e015a440 2024-03-18T14:42:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.873+0000 7f76d6ffd640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f76c006bbd0 0x7f76c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:31.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.873+0000 7f76d6ffd640 1 -- 172.21.15.6:0/3653234916 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f76d808b7b0 con 0x7f76e015a440 2024-03-18T14:42:31.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.874+0000 7f76de59c640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f76c006bbd0 0x7f76c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:31.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.874+0000 7f76de59c640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f76c006bbd0 0x7f76c006e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76d0036000 tx=0x7f76d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:31.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:31.878+0000 7f76d6ffd640 1 -- 172.21.15.6:0/3653234916 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f76d8022070 con 0x7f76e015a440 2024-03-18T14:42:32.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.071+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f76e006c920 con 0x7f76e015a440 2024-03-18T14:42:32.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.072+0000 7f76d6ffd640 1 -- 172.21.15.6:0/3653234916 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f76d8010040 con 0x7f76e015a440 2024-03-18T14:42:32.075 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:32.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f76c006bbd0 msgr2=0x7f76c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f76c006bbd0 0x7f76c006e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76d0036000 tx=0x7f76d0008030 comp rx=0 tx=0).stop 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 msgr2=0x7f76e00aaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00aaa00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009dd0 tx=0x7f76d800c490 comp rx=0 tx=0).stop 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 shutdown_connections 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f76c006bbd0 0x7f76c006e090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e015a440 0x7f76e00aaa00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e00b42a0 0x7f76e00aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 --2- 172.21.15.6:0/3653234916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e0072730 0x7f76e014b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 >> 172.21.15.6:0/3653234916 conn(0x7f76e00b1570 msgr2=0x7f76e015b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 shutdown_connections 2024-03-18T14:42:32.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.074+0000 7f76e4b5b640 1 -- 172.21.15.6:0/3653234916 wait complete. 2024-03-18T14:42:32.087 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:42:32.087 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:32.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.203+0000 7fd0fd275640 1 Processor -- start 2024-03-18T14:42:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.203+0000 7fd0fd275640 1 -- start start 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0fd275640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd0f815e170 0x7fd0f80b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0fd275640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d0d90 0x7fd0f80b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0fd275640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0f80cef50 con 0x7fd0f80d3330 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0fd275640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f80b9b90 con 0x7fd0f815e170 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0fd275640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f80cf230 con 0x7fd0f80d0d90 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f6ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd0f815e170 0x7fd0f80b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f77fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d0d90 0x7fd0f80b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f6ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd0f815e170 0x7fd0f80b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46386/0 (socket says 172.21.15.6:46386) 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f67fc640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd0f80d3330 0x7fd0f80d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:50740/0 (socket says 172.21.15.6:50740) 2024-03-18T14:42:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/2637203501 learned_addr learned my addr 172.21.15.6:0/2637203501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/2637203501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d0d90 msgr2=0x7fd0f80b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f6ffd640 1 --2- 172.21.15.6:0/2637203501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d0d90 0x7fd0f80b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/2637203501 >> v1:172.21.15.92:6789/0 conn(0x7fd0f80d3330 legacy=0x7fd0f80d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/2637203501 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f80d2b30 con 0x7fd0f815e170 2024-03-18T14:42:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.204+0000 7fd0f5ffb640 1 -- 172.21.15.6:0/2637203501 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789836251 0 0) 0x7fd0f80cef50 con 0x7fd0f80d3330 2024-03-18T14:42:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.205+0000 7fd0f6ffd640 1 --2- 172.21.15.6:0/2637203501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f815e170 0x7fd0f80b9650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009870 tx=0x7fd0f002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70d4021b9a5e6a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.205+0000 7fd0f5ffb640 1 -- 172.21.15.6:0/2637203501 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0034070 con 0x7fd0f815e170 2024-03-18T14:42:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.205+0000 7fd0f5ffb640 1 -- 172.21.15.6:0/2637203501 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0f0030840 con 0x7fd0f815e170 2024-03-18T14:42:32.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.205+0000 7fd0f5ffb640 1 -- 172.21.15.6:0/2637203501 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0030c10 con 0x7fd0f815e170 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 -- 172.21.15.6:0/2637203501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f815e170 msgr2=0x7fd0f80b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 --2- 172.21.15.6:0/2637203501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f815e170 0x7fd0f80b9650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009870 tx=0x7fd0f002b980 comp rx=0 tx=0).stop 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 -- 172.21.15.6:0/2637203501 shutdown_connections 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 --2- 172.21.15.6:0/2637203501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d0d90 0x7fd0f80b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 --2- 172.21.15.6:0/2637203501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f815e170 0x7fd0f80b9650 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 -- 172.21.15.6:0/2637203501 >> 172.21.15.6:0/2637203501 conn(0x7fd0f806c900 msgr2=0x7fd0f806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 -- 172.21.15.6:0/2637203501 shutdown_connections 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.206+0000 7fd0fd275640 1 -- 172.21.15.6:0/2637203501 wait complete. 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.207+0000 7fd0fd275640 1 Processor -- start 2024-03-18T14:42:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.207+0000 7fd0fd275640 1 -- start start 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0fd275640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f80d0d90 0x7fd0f81f8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0fd275640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d3330 0x7fd0f8161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0fd275640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0f815e170 0x7fd0f81598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0fd275640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f80ceb70 con 0x7fd0f815e170 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0fd275640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f80d2820 con 0x7fd0f80d0d90 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0fd275640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f80cecf0 con 0x7fd0f80d3330 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f80d0d90 0x7fd0f81f8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0f815e170 0x7fd0f81598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f80d0d90 0x7fd0f81f8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46398/0 (socket says 172.21.15.6:46398) 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/1145344726 learned_addr learned my addr 172.21.15.6:0/1145344726 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f67fc640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d3330 0x7fd0f8161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d3330 msgr2=0x7fd0f8161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d3330 0x7fd0f8161120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/1145344726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0f815e170 msgr2=0x7fd0f81598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0f815e170 0x7fd0f81598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 -- 172.21.15.6:0/1145344726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f8160080 con 0x7fd0f80d0d90 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0f6ffd640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f80d0d90 0x7fd0f81f8420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000c00 tx=0x7fd0f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0ef7fe640 1 -- 172.21.15.6:0/1145344726 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0034040 con 0x7fd0f80d0d90 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0ef7fe640 1 -- 172.21.15.6:0/1145344726 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0f003ab90 con 0x7fd0f80d0d90 2024-03-18T14:42:32.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.208+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f8160360 con 0x7fd0f80d0d90 2024-03-18T14:42:32.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.209+0000 7fd0ef7fe640 1 -- 172.21.15.6:0/1145344726 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0004030 con 0x7fd0f80d0d90 2024-03-18T14:42:32.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.209+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f8160920 con 0x7fd0f80d0d90 2024-03-18T14:42:32.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.209+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f808afe0 con 0x7fd0f80d0d90 2024-03-18T14:42:32.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.210+0000 7fd0ef7fe640 1 -- 172.21.15.6:0/1145344726 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd0f004b060 con 0x7fd0f80d0d90 2024-03-18T14:42:32.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.210+0000 7fd0ef7fe640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd0d806bbd0 0x7fd0d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.211+0000 7fd0ef7fe640 1 -- 172.21.15.6:0/1145344726 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd0f00acaf0 con 0x7fd0f80d0d90 2024-03-18T14:42:32.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.211+0000 7fd0f67fc640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd0d806bbd0 0x7fd0d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.211+0000 7fd0f67fc640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd0d806bbd0 0x7fd0d806e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd0f81f7970 tx=0x7fd0e0007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.215+0000 7fd0ef7fe640 1 -- 172.21.15.6:0/1145344726 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd0f003f070 con 0x7fd0f80d0d90 2024-03-18T14:42:32.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.401+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0f804f620 con 0x7fd0f80d0d90 2024-03-18T14:42:32.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.402+0000 7fd0ef7fe640 1 -- 172.21.15.6:0/1145344726 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fd0f007aaf0 con 0x7fd0f80d0d90 2024-03-18T14:42:32.405 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:32.405 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:32.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.404+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd0d806bbd0 msgr2=0x7fd0d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.404+0000 7fd0fd275640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd0d806bbd0 0x7fd0d806e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd0f81f7970 tx=0x7fd0e0007d50 comp rx=0 tx=0).stop 2024-03-18T14:42:32.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.404+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f80d0d90 msgr2=0x7fd0f81f8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.404+0000 7fd0fd275640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f80d0d90 0x7fd0f81f8420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000c00 tx=0x7fd0f00046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 shutdown_connections 2024-03-18T14:42:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd0d806bbd0 0x7fd0d806e090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0f815e170 0x7fd0f81598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0f80d3330 0x7fd0f8161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 --2- 172.21.15.6:0/1145344726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0f80d0d90 0x7fd0f81f8420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 >> 172.21.15.6:0/1145344726 conn(0x7fd0f806c900 msgr2=0x7fd0f80c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 shutdown_connections 2024-03-18T14:42:32.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.405+0000 7fd0fd275640 1 -- 172.21.15.6:0/1145344726 wait complete. 2024-03-18T14:42:32.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:32.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.520+0000 7f4f405f4640 1 Processor -- start 2024-03-18T14:42:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.520+0000 7f4f405f4640 1 -- start start 2024-03-18T14:42:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f405f4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f380cf850 0x7f4f380cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f405f4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cd8f0 0x7f4f380b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f405f4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f380ce590 con 0x7f4f3815ac60 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f405f4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4f380cdd40 con 0x7f4f380cf850 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f405f4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4f380cf3c0 con 0x7f4f380cd8f0 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f3db8f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f380cf850 0x7f4f380cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f3e390640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f3815ac60 0x7f4f380cc170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56052/0 (socket says 172.21.15.6:56052) 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f3e390640 1 -- 172.21.15.6:0/1350029371 learned_addr learned my addr 172.21.15.6:0/1350029371 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f3db8f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f380cf850 0x7f4f380cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37076/0 (socket says 172.21.15.6:37076) 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.521+0000 7f4f3eb91640 1 --2- 172.21.15.6:0/1350029371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cd8f0 0x7f4f380b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.522+0000 7f4f3db8f640 1 -- 172.21.15.6:0/1350029371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cd8f0 msgr2=0x7f4f380b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.522+0000 7f4f3db8f640 1 --2- 172.21.15.6:0/1350029371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cd8f0 0x7f4f380b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.522+0000 7f4f3db8f640 1 -- 172.21.15.6:0/1350029371 >> v1:172.21.15.6:6789/0 conn(0x7f4f3815ac60 legacy=0x7f4f380cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.522+0000 7f4f3db8f640 1 -- 172.21.15.6:0/1350029371 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f380ce830 con 0x7f4f380cf850 2024-03-18T14:42:32.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.522+0000 7f4f3d38e640 1 -- 172.21.15.6:0/1350029371 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 495446185 0 0) 0x7f4f380ce590 con 0x7f4f3815ac60 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.522+0000 7f4f3db8f640 1 --2- 172.21.15.6:0/1350029371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f380cf850 0x7f4f380cd3b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00ba40 tx=0x7f4f2c02e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83de8a58ff21db89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f3d38e640 1 -- 172.21.15.6:0/1350029371 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c03b070 con 0x7f4f380cf850 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f3d38e640 1 -- 172.21.15.6:0/1350029371 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f2c02ee30 con 0x7f4f380cf850 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f3d38e640 1 -- 172.21.15.6:0/1350029371 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c036640 con 0x7f4f380cf850 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 -- 172.21.15.6:0/1350029371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f380cf850 msgr2=0x7f4f380cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 --2- 172.21.15.6:0/1350029371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f380cf850 0x7f4f380cd3b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00ba40 tx=0x7f4f2c02e080 comp rx=0 tx=0).stop 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 -- 172.21.15.6:0/1350029371 shutdown_connections 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 --2- 172.21.15.6:0/1350029371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cd8f0 0x7f4f380b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 --2- 172.21.15.6:0/1350029371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f380cf850 0x7f4f380cd3b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 -- 172.21.15.6:0/1350029371 >> 172.21.15.6:0/1350029371 conn(0x7f4f38099a50 msgr2=0x7f4f380d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 -- 172.21.15.6:0/1350029371 shutdown_connections 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.523+0000 7f4f405f4640 1 -- 172.21.15.6:0/1350029371 wait complete. 2024-03-18T14:42:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.524+0000 7f4f405f4640 1 Processor -- start 2024-03-18T14:42:32.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.524+0000 7f4f405f4640 1 -- start start 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f405f4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f380cd8f0 0x7f4f38156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f405f4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cf850 0x7f4f3806f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f405f4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3815ac60 0x7f4f3806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f405f4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f380cf050 con 0x7f4f3815ac60 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f405f4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4f380cead0 con 0x7f4f380cd8f0 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f405f4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3815b040 con 0x7f4f380cf850 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f3eb91640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3815ac60 0x7f4f3806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f3eb91640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3815ac60 0x7f4f3806feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37082/0 (socket says 172.21.15.6:37082) 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f3eb91640 1 -- 172.21.15.6:0/2724595270 learned_addr learned my addr 172.21.15.6:0/2724595270 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f3e390640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f380cd8f0 0x7f4f38156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.525+0000 7f4f3db8f640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cf850 0x7f4f3806f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f3eb91640 1 -- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cf850 msgr2=0x7f4f3806f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f3eb91640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cf850 0x7f4f3806f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f3eb91640 1 -- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f380cd8f0 msgr2=0x7f4f38156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f3eb91640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f380cd8f0 0x7f4f38156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f3eb91640 1 -- 172.21.15.6:0/2724595270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f38157450 con 0x7f4f3815ac60 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f3eb91640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3815ac60 0x7f4f3806feb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000e930 tx=0x7f4f3000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f36ffd640 1 -- 172.21.15.6:0/2724595270 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30017070 con 0x7f4f3815ac60 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f36ffd640 1 -- 172.21.15.6:0/2724595270 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3000c980 con 0x7f4f3815ac60 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f36ffd640 1 -- 172.21.15.6:0/2724595270 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000cd70 con 0x7f4f3815ac60 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f381ef800 con 0x7f4f3815ac60 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.526+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3815da10 con 0x7f4f3815ac60 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.527+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f380751b0 con 0x7f4f3815ac60 2024-03-18T14:42:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.527+0000 7f4f36ffd640 1 -- 172.21.15.6:0/2724595270 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4f3001a070 con 0x7f4f3815ac60 2024-03-18T14:42:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.528+0000 7f4f36ffd640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f2406bbd0 0x7f4f2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.528+0000 7f4f36ffd640 1 -- 172.21.15.6:0/2724595270 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4f3008add0 con 0x7f4f3815ac60 2024-03-18T14:42:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.528+0000 7f4f3e390640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f2406bbd0 0x7f4f2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.528+0000 7f4f3e390640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f2406bbd0 0x7f4f2406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f280012e0 tx=0x7f4f28001630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.532+0000 7f4f36ffd640 1 -- 172.21.15.6:0/2724595270 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4f30058f30 con 0x7f4f3815ac60 2024-03-18T14:42:32.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.717+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f380b19f0 con 0x7f4f3815ac60 2024-03-18T14:42:32.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.717+0000 7f4f36ffd640 1 -- 172.21.15.6:0/2724595270 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f4f3005cfd0 con 0x7f4f3815ac60 2024-03-18T14:42:32.720 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f2406bbd0 msgr2=0x7f4f2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f2406bbd0 0x7f4f2406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f280012e0 tx=0x7f4f28001630 comp rx=0 tx=0).stop 2024-03-18T14:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3815ac60 msgr2=0x7f4f3806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3815ac60 0x7f4f3806feb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000e930 tx=0x7f4f3000edf0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 shutdown_connections 2024-03-18T14:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f2406bbd0 0x7f4f2406e090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3815ac60 0x7f4f3806feb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f380cf850 0x7f4f3806f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 --2- 172.21.15.6:0/2724595270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f380cd8f0 0x7f4f38156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 >> 172.21.15.6:0/2724595270 conn(0x7f4f38099a50 msgr2=0x7f4f380cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:32.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 shutdown_connections 2024-03-18T14:42:32.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.719+0000 7f4f405f4640 1 -- 172.21.15.6:0/2724595270 wait complete. 2024-03-18T14:42:32.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:32.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.834+0000 7f0b23aff640 1 Processor -- start 2024-03-18T14:42:32.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.834+0000 7f0b23aff640 1 -- start start 2024-03-18T14:42:32.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b23aff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b1c15e030 0x7f0b1c0cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b23aff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1c0ce040 con 0x7f0b1c0afac0 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b23aff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1c0cdda0 con 0x7f0b1c06b8d0 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b23aff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0b1c15e410 con 0x7f0b1c15e030 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b2189b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b1c15e030 0x7f0b1c0cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b2109a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b1c0afac0 0x7f0b1c06b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56068/0 (socket says 172.21.15.6:56068) 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b2109a640 1 -- 172.21.15.6:0/3418122674 learned_addr learned my addr 172.21.15.6:0/3418122674 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b2189b640 1 -- 172.21.15.6:0/3418122674 >> v1:172.21.15.92:6789/0 conn(0x7f0b1c06b8d0 legacy=0x7f0b1c0cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b2189b640 1 -- 172.21.15.6:0/3418122674 >> v1:172.21.15.6:6789/0 conn(0x7f0b1c0afac0 legacy=0x7f0b1c06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.835+0000 7f0b2189b640 1 -- 172.21.15.6:0/3418122674 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c0d0870 con 0x7f0b1c15e030 2024-03-18T14:42:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.836+0000 7f0b2189b640 1 --2- 172.21.15.6:0/3418122674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c15e030 0x7f0b1c0cd860 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0b180097a0 tx=0x7f0b1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a084752a2e62e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.836+0000 7f0b20899640 1 -- 172.21.15.6:0/3418122674 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18035020 con 0x7f0b1c15e030 2024-03-18T14:42:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.836+0000 7f0b20899640 1 -- 172.21.15.6:0/3418122674 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b18031830 con 0x7f0b1c15e030 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.836+0000 7f0b20899640 1 -- 172.21.15.6:0/3418122674 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18031c00 con 0x7f0b1c15e030 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.837+0000 7f0b23aff640 1 -- 172.21.15.6:0/3418122674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c15e030 msgr2=0x7f0b1c0cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.837+0000 7f0b23aff640 1 --2- 172.21.15.6:0/3418122674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c15e030 0x7f0b1c0cd860 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0b180097a0 tx=0x7f0b1802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.837+0000 7f0b23aff640 1 -- 172.21.15.6:0/3418122674 shutdown_connections 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.837+0000 7f0b23aff640 1 --2- 172.21.15.6:0/3418122674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c15e030 0x7f0b1c0cd860 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.837+0000 7f0b23aff640 1 -- 172.21.15.6:0/3418122674 >> 172.21.15.6:0/3418122674 conn(0x7f0b1c099a50 msgr2=0x7f0b1c099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.837+0000 7f0b23aff640 1 -- 172.21.15.6:0/3418122674 shutdown_connections 2024-03-18T14:42:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.837+0000 7f0b23aff640 1 -- 172.21.15.6:0/3418122674 wait complete. 2024-03-18T14:42:32.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.838+0000 7f0b23aff640 1 Processor -- start 2024-03-18T14:42:32.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.838+0000 7f0b23aff640 1 -- start start 2024-03-18T14:42:32.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b1c06b8d0 0x7f0b1c155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c0afac0 0x7f0b1c14bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1c15e030 0x7f0b1c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c0d0e30 con 0x7f0b1c0afac0 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c0d11a0 con 0x7f0b1c06b8d0 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0b1c0d1630 con 0x7f0b1c15e030 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c0afac0 0x7f0b1c14bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c0afac0 0x7f0b1c14bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37100/0 (socket says 172.21.15.6:37100) 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 -- 172.21.15.6:0/2267396333 learned_addr learned my addr 172.21.15.6:0/2267396333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2189b640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b1c06b8d0 0x7f0b1c155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2209c640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1c15e030 0x7f0b1c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 -- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1c15e030 msgr2=0x7f0b1c0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1c15e030 0x7f0b1c0c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 -- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b1c06b8d0 msgr2=0x7f0b1c155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b1c06b8d0 0x7f0b1c155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:32.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 -- 172.21.15.6:0/2267396333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c0c4d00 con 0x7f0b1c0afac0 2024-03-18T14:42:32.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b2109a640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c0afac0 0x7f0b1c14bf10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c4b0 tx=0x7f0b0c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b127fc640 1 -- 172.21.15.6:0/2267396333 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c017070 con 0x7f0b1c0afac0 2024-03-18T14:42:32.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b127fc640 1 -- 172.21.15.6:0/2267396333 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0c005ce0 con 0x7f0b1c0afac0 2024-03-18T14:42:32.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1c0c4fe0 con 0x7f0b1c0afac0 2024-03-18T14:42:32.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b127fc640 1 -- 172.21.15.6:0/2267396333 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c007270 con 0x7f0b1c0afac0 2024-03-18T14:42:32.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.839+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1c0c55a0 con 0x7f0b1c0afac0 2024-03-18T14:42:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.840+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b1c0751b0 con 0x7f0b1c0afac0 2024-03-18T14:42:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.841+0000 7f0b127fc640 1 -- 172.21.15.6:0/2267396333 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0b0c01a070 con 0x7f0b1c0afac0 2024-03-18T14:42:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.841+0000 7f0b127fc640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b0006bbd0 0x7f0b0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.841+0000 7f0b127fc640 1 -- 172.21.15.6:0/2267396333 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0b0c08a020 con 0x7f0b1c0afac0 2024-03-18T14:42:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.841+0000 7f0b2189b640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b0006bbd0 0x7f0b0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.842+0000 7f0b2189b640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b0006bbd0 0x7f0b0006e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0b18036000 tx=0x7f0b1803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:32.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:32.845+0000 7f0b127fc640 1 -- 172.21.15.6:0/2267396333 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0b1c0ce040 con 0x7f0b1c0afac0 2024-03-18T14:42:33.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.026+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0b1c06ed90 con 0x7f0b1c0afac0 2024-03-18T14:42:33.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.026+0000 7f0b127fc640 1 -- 172.21.15.6:0/2267396333 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f0b0c058130 con 0x7f0b1c0afac0 2024-03-18T14:42:33.029 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b0006bbd0 msgr2=0x7f0b0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b0006bbd0 0x7f0b0006e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0b18036000 tx=0x7f0b1803c040 comp rx=0 tx=0).stop 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c0afac0 msgr2=0x7f0b1c14bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c0afac0 0x7f0b1c14bf10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00c4b0 tx=0x7f0b0c00c970 comp rx=0 tx=0).stop 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 shutdown_connections 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0b0006bbd0 0x7f0b0006e090 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1c15e030 0x7f0b1c0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1c0afac0 0x7f0b1c14bf10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 --2- 172.21.15.6:0/2267396333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b1c06b8d0 0x7f0b1c155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 >> 172.21.15.6:0/2267396333 conn(0x7f0b1c099a50 msgr2=0x7f0b1c0aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 shutdown_connections 2024-03-18T14:42:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.028+0000 7f0b23aff640 1 -- 172.21.15.6:0/2267396333 wait complete. 2024-03-18T14:42:33.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:33.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.143+0000 7fd650953640 1 Processor -- start 2024-03-18T14:42:33.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.143+0000 7fd650953640 1 -- start start 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd650953640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd64c099f20 0x7fd64c0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd650953640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c0cf6a0 0x7fd64c0ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd650953640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd64c09a300 0x7fd64c0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd650953640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd64c0cdb70 con 0x7fd64c09a300 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd650953640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd64c0b1560 con 0x7fd64c099f20 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd650953640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd64c0b1800 con 0x7fd64c0cf6a0 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd64c09a300 0x7fd64c0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd64c09a300 0x7fd64c0b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37104/0 (socket says 172.21.15.6:37104) 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 -- 172.21.15.6:0/384520590 learned_addr learned my addr 172.21.15.6:0/384520590 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64a59c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd64c099f20 0x7fd64c0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd649d9b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c0cf6a0 0x7fd64c0ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 -- 172.21.15.6:0/384520590 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c0cf6a0 msgr2=0x7fd64c0ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 --2- 172.21.15.6:0/384520590 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c0cf6a0 0x7fd64c0ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 -- 172.21.15.6:0/384520590 >> v2:172.21.15.92:3300/0 conn(0x7fd64c099f20 msgr2=0x7fd64c0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:33.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 --2- 172.21.15.6:0/384520590 >> v2:172.21.15.92:3300/0 conn(0x7fd64c099f20 0x7fd64c0cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64ad9d640 1 -- 172.21.15.6:0/384520590 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c0b1ae0 con 0x7fd64c09a300 2024-03-18T14:42:33.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.144+0000 7fd64a59c640 1 --2- 172.21.15.6:0/384520590 >> v2:172.21.15.92:3300/0 conn(0x7fd64c099f20 0x7fd64c0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.145+0000 7fd64ad9d640 1 --2- 172.21.15.6:0/384520590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c0b22e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd64c067f90 tx=0x7fd64402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219a44eda7de1aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.145+0000 7fd64959a640 1 -- 172.21.15.6:0/384520590 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64402c950 con 0x7fd64c09a300 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.145+0000 7fd64959a640 1 -- 172.21.15.6:0/384520590 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64402caf0 con 0x7fd64c09a300 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.145+0000 7fd64959a640 1 -- 172.21.15.6:0/384520590 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6440343f0 con 0x7fd64c09a300 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.145+0000 7fd650953640 1 -- 172.21.15.6:0/384520590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 msgr2=0x7fd64c0b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.145+0000 7fd650953640 1 --2- 172.21.15.6:0/384520590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c0b22e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd64c067f90 tx=0x7fd64402bdb0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.146+0000 7fd650953640 1 -- 172.21.15.6:0/384520590 shutdown_connections 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.146+0000 7fd650953640 1 --2- 172.21.15.6:0/384520590 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c0b22e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.146+0000 7fd650953640 1 --2- 172.21.15.6:0/384520590 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c0cf6a0 0x7fd64c0ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.146+0000 7fd650953640 1 --2- 172.21.15.6:0/384520590 >> v2:172.21.15.92:3300/0 conn(0x7fd64c099f20 0x7fd64c0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.146+0000 7fd650953640 1 -- 172.21.15.6:0/384520590 >> 172.21.15.6:0/384520590 conn(0x7fd64c0b35c0 msgr2=0x7fd64c0b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.146+0000 7fd650953640 1 -- 172.21.15.6:0/384520590 shutdown_connections 2024-03-18T14:42:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.146+0000 7fd650953640 1 -- 172.21.15.6:0/384520590 wait complete. 2024-03-18T14:42:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.147+0000 7fd650953640 1 Processor -- start 2024-03-18T14:42:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.147+0000 7fd650953640 1 -- start start 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c099f20 0x7fd64c1f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c06fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd64c0cf6a0 0x7fd64c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c0cd900 con 0x7fd64c09a300 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c0cd780 con 0x7fd64c0cf6a0 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd64c0b2820 con 0x7fd64c099f20 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c06fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c06fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37120/0 (socket says 172.21.15.6:37120) 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 -- 172.21.15.6:0/3788037336 learned_addr learned my addr 172.21.15.6:0/3788037336 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd64a59c640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c099f20 0x7fd64c1f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd64ad9d640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd64c0cf6a0 0x7fd64c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 -- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c099f20 msgr2=0x7fd64c1f4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c099f20 0x7fd64c1f4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 -- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd64c0cf6a0 msgr2=0x7fd64c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd64c0cf6a0 0x7fd64c070280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 -- 172.21.15.6:0/3788037336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c1f2270 con 0x7fd64c09a300 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd649d9b640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c06fd40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd64000e8a0 tx=0x7fd64000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd63affd640 1 -- 172.21.15.6:0/3788037336 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640017070 con 0x7fd64c09a300 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd63affd640 1 -- 172.21.15.6:0/3788037336 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64000c940 con 0x7fd64c09a300 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64c1f2550 con 0x7fd64c09a300 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd63affd640 1 -- 172.21.15.6:0/3788037336 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64000cd30 con 0x7fd64c09a300 2024-03-18T14:42:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.148+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64c1f1f60 con 0x7fd64c09a300 2024-03-18T14:42:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.150+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64c0751b0 con 0x7fd64c09a300 2024-03-18T14:42:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.150+0000 7fd63affd640 1 -- 172.21.15.6:0/3788037336 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd64001a070 con 0x7fd64c09a300 2024-03-18T14:42:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.150+0000 7fd63affd640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd62c06bbd0 0x7fd62c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.150+0000 7fd63affd640 1 -- 172.21.15.6:0/3788037336 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd64008b000 con 0x7fd64c09a300 2024-03-18T14:42:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.150+0000 7fd64a59c640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd62c06bbd0 0x7fd62c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.151+0000 7fd64a59c640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd62c06bbd0 0x7fd62c06e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009560 tx=0x7fd63c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.154+0000 7fd63affd640 1 -- 172.21.15.6:0/3788037336 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd640059020 con 0x7fd64c09a300 2024-03-18T14:42:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.335+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd64c0ce2c0 con 0x7fd64c09a300 2024-03-18T14:42:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.335+0000 7fd63affd640 1 -- 172.21.15.6:0/3788037336 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fd64005d0c0 con 0x7fd64c09a300 2024-03-18T14:42:33.338 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd62c06bbd0 msgr2=0x7fd62c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd62c06bbd0 0x7fd62c06e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009560 tx=0x7fd63c009190 comp rx=0 tx=0).stop 2024-03-18T14:42:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 msgr2=0x7fd64c06fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c06fd40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd64000e8a0 tx=0x7fd64000ed60 comp rx=0 tx=0).stop 2024-03-18T14:42:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 shutdown_connections 2024-03-18T14:42:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd62c06bbd0 0x7fd62c06e090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd64c0cf6a0 0x7fd64c070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd64c09a300 0x7fd64c06fd40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 --2- 172.21.15.6:0/3788037336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd64c099f20 0x7fd64c1f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 >> 172.21.15.6:0/3788037336 conn(0x7fd64c0b35c0 msgr2=0x7fd64c0aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 shutdown_connections 2024-03-18T14:42:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.337+0000 7fd650953640 1 -- 172.21.15.6:0/3788037336 wait complete. 2024-03-18T14:42:33.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.453+0000 7f193cbf4640 1 Processor -- start 2024-03-18T14:42:33.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.453+0000 7f193cbf4640 1 -- start start 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f193cbf4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f19381477a0 0x7f19380ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f193cbf4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f19380cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f193cbf4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19380d0230 0x7f1938071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f193cbf4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1938072e00 con 0x7f19380d0230 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f193cbf4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1938072b20 con 0x7f19381477a0 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f193cbf4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f193806e700 con 0x7f19380ad6a0 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19380d0230 0x7f1938071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19380d0230 0x7f1938071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37130/0 (socket says 172.21.15.6:37130) 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 -- 172.21.15.6:0/911921226 learned_addr learned my addr 172.21.15.6:0/911921226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f192ffff640 1 --2- 172.21.15.6:0/911921226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f19380cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f193659c640 1 --2- 172.21.15.6:0/911921226 >> v2:172.21.15.92:3300/0 conn(0x7f19381477a0 0x7f19380ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 -- 172.21.15.6:0/911921226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 msgr2=0x7f19380cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:33.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 --2- 172.21.15.6:0/911921226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f19380cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 -- 172.21.15.6:0/911921226 >> v2:172.21.15.92:3300/0 conn(0x7f19381477a0 msgr2=0x7f19380ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 --2- 172.21.15.6:0/911921226 >> v2:172.21.15.92:3300/0 conn(0x7f19381477a0 0x7f19380ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f1936d9d640 1 -- 172.21.15.6:0/911921226 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193806e9e0 con 0x7f19380d0230 2024-03-18T14:42:33.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.454+0000 7f192ffff640 1 --2- 172.21.15.6:0/911921226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f19380cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.455+0000 7f1936d9d640 1 --2- 172.21.15.6:0/911921226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19380d0230 0x7f1938071e60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f19380730b0 tx=0x7f193002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5bfc5a10bc6013 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.455+0000 7f1935d9b640 1 -- 172.21.15.6:0/911921226 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193002c9e0 con 0x7f19380d0230 2024-03-18T14:42:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.455+0000 7f1935d9b640 1 -- 172.21.15.6:0/911921226 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f193002cb80 con 0x7f19380d0230 2024-03-18T14:42:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.455+0000 7f1935d9b640 1 -- 172.21.15.6:0/911921226 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19300343f0 con 0x7f19380d0230 2024-03-18T14:42:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.455+0000 7f193cbf4640 1 -- 172.21.15.6:0/911921226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19380d0230 msgr2=0x7f1938071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.455+0000 7f193cbf4640 1 --2- 172.21.15.6:0/911921226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19380d0230 0x7f1938071e60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f19380730b0 tx=0x7f193002be40 comp rx=0 tx=0).stop 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.456+0000 7f193cbf4640 1 -- 172.21.15.6:0/911921226 shutdown_connections 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.456+0000 7f193cbf4640 1 --2- 172.21.15.6:0/911921226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19380d0230 0x7f1938071e60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.456+0000 7f193cbf4640 1 --2- 172.21.15.6:0/911921226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f19380cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.456+0000 7f193cbf4640 1 --2- 172.21.15.6:0/911921226 >> v2:172.21.15.92:3300/0 conn(0x7f19381477a0 0x7f19380ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.456+0000 7f193cbf4640 1 -- 172.21.15.6:0/911921226 >> 172.21.15.6:0/911921226 conn(0x7f193806b1c0 msgr2=0x7f193806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.456+0000 7f193cbf4640 1 -- 172.21.15.6:0/911921226 shutdown_connections 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.456+0000 7f193cbf4640 1 -- 172.21.15.6:0/911921226 wait complete. 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.457+0000 7f193cbf4640 1 Processor -- start 2024-03-18T14:42:33.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.457+0000 7f193cbf4640 1 -- start start 2024-03-18T14:42:33.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f193813b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19380d0230 0x7f193813c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19381477a0 0x7f19380b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1938147d00 con 0x7f19381477a0 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1938147b80 con 0x7f19380d0230 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f19380723a0 con 0x7f19380ad6a0 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19381477a0 0x7f19380b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19381477a0 0x7f19380b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37136/0 (socket says 172.21.15.6:37136) 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 -- 172.21.15.6:0/3267189669 learned_addr learned my addr 172.21.15.6:0/3267189669 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193659c640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f193813b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f192ffff640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19380d0230 0x7f193813c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 -- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 msgr2=0x7f193813b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f193813b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 -- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19380d0230 msgr2=0x7f193813c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19380d0230 0x7f193813c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 -- 172.21.15.6:0/3267189669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193813ee60 con 0x7f19381477a0 2024-03-18T14:42:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f1936d9d640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19381477a0 0x7f19380b0a80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1930009f40 tx=0x7f1930004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f192effd640 1 -- 172.21.15.6:0/3267189669 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930043020 con 0x7f19381477a0 2024-03-18T14:42:33.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f192effd640 1 -- 172.21.15.6:0/3267189669 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f193003e450 con 0x7f19381477a0 2024-03-18T14:42:33.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f192effd640 1 -- 172.21.15.6:0/3267189669 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930009510 con 0x7f19381477a0 2024-03-18T14:42:33.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193813f140 con 0x7f19381477a0 2024-03-18T14:42:33.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.458+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19381401f0 con 0x7f19381477a0 2024-03-18T14:42:33.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.460+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19380751b0 con 0x7f19381477a0 2024-03-18T14:42:33.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.460+0000 7f192effd640 1 -- 172.21.15.6:0/3267189669 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1930046020 con 0x7f19381477a0 2024-03-18T14:42:33.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.460+0000 7f192effd640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f191806bbd0 0x7f191806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.460+0000 7f192effd640 1 -- 172.21.15.6:0/3267189669 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f19300ad3e0 con 0x7f19381477a0 2024-03-18T14:42:33.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.461+0000 7f193659c640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f191806bbd0 0x7f191806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.461+0000 7f193659c640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f191806bbd0 0x7f191806e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1928009aa0 tx=0x7f1928009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.465+0000 7f192effd640 1 -- 172.21.15.6:0/3267189669 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f193004d030 con 0x7f19381477a0 2024-03-18T14:42:33.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.646+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f193806e3b0 con 0x7f19381477a0 2024-03-18T14:42:33.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.646+0000 7f192effd640 1 -- 172.21.15.6:0/3267189669 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f193007b480 con 0x7f19381477a0 2024-03-18T14:42:33.649 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f191806bbd0 msgr2=0x7f191806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f191806bbd0 0x7f191806e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1928009aa0 tx=0x7f1928009190 comp rx=0 tx=0).stop 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19381477a0 msgr2=0x7f19380b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19381477a0 0x7f19380b0a80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1930009f40 tx=0x7f1930004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 shutdown_connections 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f191806bbd0 0x7f191806e090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19381477a0 0x7f19380b0a80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19380d0230 0x7f193813c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 --2- 172.21.15.6:0/3267189669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19380ad6a0 0x7f193813b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 >> 172.21.15.6:0/3267189669 conn(0x7f193806b1c0 msgr2=0x7f1938160290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:33.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 shutdown_connections 2024-03-18T14:42:33.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.648+0000 7f193cbf4640 1 -- 172.21.15.6:0/3267189669 wait complete. 2024-03-18T14:42:33.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:33.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.763+0000 7f18a0759640 1 Processor -- start 2024-03-18T14:42:33.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.763+0000 7f18a0759640 1 -- start start 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f18a0759640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f18a0759640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f18980cda00 0x7f189806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f18a0759640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18980b12b0 con 0x7f189806b1c0 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f18a0759640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f18980d18a0 con 0x7f18980cda00 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f18a0759640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f189806bb40 con 0x7f18980b0ed0 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f189dcf4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f189806b1c0 0x7f189815f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49924/0 (socket says 172.21.15.6:49924) 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f189dcf4640 1 -- 172.21.15.6:0/4222811457 learned_addr learned my addr 172.21.15.6:0/4222811457 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f189e4f5640 1 --2- 172.21.15.6:0/4222811457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f189ecf6640 1 --2- 172.21.15.6:0/4222811457 >> v2:172.21.15.92:3300/0 conn(0x7f18980cda00 0x7f189806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.764+0000 7f189d4f3640 1 -- 172.21.15.6:0/4222811457 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3902534805 0 0) 0x7f1888001110 con 0x7f189806b1c0 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.765+0000 7f189d4f3640 1 -- 172.21.15.6:0/4222811457 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1884003520 con 0x7f189806b1c0 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.765+0000 7f189e4f5640 1 -- 172.21.15.6:0/4222811457 >> v2:172.21.15.92:3300/0 conn(0x7f18980cda00 msgr2=0x7f189806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.765+0000 7f189e4f5640 1 --2- 172.21.15.6:0/4222811457 >> v2:172.21.15.92:3300/0 conn(0x7f18980cda00 0x7f189806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.765+0000 7f189e4f5640 1 -- 172.21.15.6:0/4222811457 >> v1:172.21.15.6:6789/0 conn(0x7f189806b1c0 legacy=0x7f189815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.765+0000 7f189e4f5640 1 -- 172.21.15.6:0/4222811457 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18980d0c90 con 0x7f18980b0ed0 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.765+0000 7f189ecf6640 1 --2- 172.21.15.6:0/4222811457 >> v2:172.21.15.92:3300/0 conn(0x7f18980cda00 0x7f189806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.765+0000 7f189e4f5640 1 --2- 172.21.15.6:0/4222811457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980b1e60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f18900097a0 tx=0x7f189002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de6c3673ab288e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f189d4f3640 1 -- 172.21.15.6:0/4222811457 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890035020 con 0x7f18980b0ed0 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f189d4f3640 1 -- 172.21.15.6:0/4222811457 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1890031830 con 0x7f18980b0ed0 2024-03-18T14:42:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f189d4f3640 1 -- 172.21.15.6:0/4222811457 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890031c00 con 0x7f18980b0ed0 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 -- 172.21.15.6:0/4222811457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 msgr2=0x7f18980b1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 --2- 172.21.15.6:0/4222811457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980b1e60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f18900097a0 tx=0x7f189002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 -- 172.21.15.6:0/4222811457 shutdown_connections 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 --2- 172.21.15.6:0/4222811457 >> v2:172.21.15.92:3300/0 conn(0x7f18980cda00 0x7f189806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 --2- 172.21.15.6:0/4222811457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980b1e60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 -- 172.21.15.6:0/4222811457 >> 172.21.15.6:0/4222811457 conn(0x7f1898099a50 msgr2=0x7f1898099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 -- 172.21.15.6:0/4222811457 shutdown_connections 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.766+0000 7f18a0759640 1 -- 172.21.15.6:0/4222811457 wait complete. 2024-03-18T14:42:33.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.767+0000 7f18a0759640 1 Processor -- start 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.767+0000 7f18a0759640 1 -- start start 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f18a0759640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189806b1c0 0x7f189815f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f18a0759640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f18a0759640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980cda00 0x7f1898151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f18a0759640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18980d1530 con 0x7f18980cda00 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f18a0759640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f18980d1250 con 0x7f189806b1c0 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f18a0759640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f18980d08f0 con 0x7f18980b0ed0 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f189ecf6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980cda00 0x7f1898151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f189ecf6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980cda00 0x7f1898151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36764/0 (socket says 172.21.15.6:36764) 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f189ecf6640 1 -- 172.21.15.6:0/3857330142 learned_addr learned my addr 172.21.15.6:0/3857330142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f189dcf4640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.768+0000 7f189e4f5640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189806b1c0 0x7f189815f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f189ecf6640 1 -- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 msgr2=0x7f18980cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f189ecf6640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f189ecf6640 1 -- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189806b1c0 msgr2=0x7f189815f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f189ecf6640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189806b1c0 0x7f189815f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f189ecf6640 1 -- 172.21.15.6:0/3857330142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18980ae260 con 0x7f18980cda00 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f189ecf6640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980cda00 0x7f1898151f10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f189400ef90 tx=0x7f1894009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f188effd640 1 -- 172.21.15.6:0/3857330142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894018020 con 0x7f18980cda00 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f188effd640 1 -- 172.21.15.6:0/3857330142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1894014480 con 0x7f18980cda00 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18980ae540 con 0x7f18980cda00 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f188effd640 1 -- 172.21.15.6:0/3857330142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894014870 con 0x7f18980cda00 2024-03-18T14:42:33.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.769+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18980aeb00 con 0x7f18980cda00 2024-03-18T14:42:33.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.770+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f189804f5b0 con 0x7f18980cda00 2024-03-18T14:42:33.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.770+0000 7f188effd640 1 -- 172.21.15.6:0/3857330142 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1894005d10 con 0x7f18980cda00 2024-03-18T14:42:33.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.771+0000 7f188effd640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f187c06bbd0 0x7f187c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:33.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.771+0000 7f188effd640 1 -- 172.21.15.6:0/3857330142 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f189408bb00 con 0x7f18980cda00 2024-03-18T14:42:33.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.771+0000 7f189e4f5640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f187c06bbd0 0x7f187c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:33.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.771+0000 7f189e4f5640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f187c06bbd0 0x7f187c06e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1890036000 tx=0x7f1890008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:33.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.775+0000 7f188effd640 1 -- 172.21.15.6:0/3857330142 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1894022030 con 0x7f18980cda00 2024-03-18T14:42:33.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.970+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1898072d50 con 0x7f18980cda00 2024-03-18T14:42:33.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.970+0000 7f188effd640 1 -- 172.21.15.6:0/3857330142 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f1894010040 con 0x7f18980cda00 2024-03-18T14:42:33.973 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:33.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f187c06bbd0 msgr2=0x7f187c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f187c06bbd0 0x7f187c06e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1890036000 tx=0x7f1890008030 comp rx=0 tx=0).stop 2024-03-18T14:42:33.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980cda00 msgr2=0x7f1898151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:33.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980cda00 0x7f1898151f10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f189400ef90 tx=0x7f1894009f90 comp rx=0 tx=0).stop 2024-03-18T14:42:33.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 shutdown_connections 2024-03-18T14:42:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f187c06bbd0 0x7f187c06e090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980cda00 0x7f1898151f10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f18980b0ed0 0x7f18980cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 --2- 172.21.15.6:0/3857330142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189806b1c0 0x7f189815f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.972+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 >> 172.21.15.6:0/3857330142 conn(0x7f1898099a50 msgr2=0x7f18980b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.973+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 shutdown_connections 2024-03-18T14:42:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:33.973+0000 7f18a0759640 1 -- 172.21.15.6:0/3857330142 wait complete. 2024-03-18T14:42:33.985 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi092.front.sepia.ceph.com... 2024-03-18T14:42:33.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:33.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-18T14:42:34.016 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:34.016 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:42:34.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:34.017 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T14:42:34.080 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:42:34.080 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:34.096 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:42:34.096 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:34.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:34.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.290+0000 7f1eac19e640 1 Processor -- start 2024-03-18T14:42:34.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.290+0000 7f1eac19e640 1 -- start start 2024-03-18T14:42:34.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.291+0000 7f1eac19e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea415bd40 0x7f1ea40d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.291+0000 7f1eac19e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea40cc870 con 0x7f1ea40cfea0 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.291+0000 7f1eac19e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea40d0500 con 0x7f1ea40d1d90 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.291+0000 7f1eac19e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea40d07e0 con 0x7f1ea415bd40 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.291+0000 7f1eaa73b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1ea40cfea0 0x7f1ea40cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49928/0 (socket says 172.21.15.6:49928) 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.291+0000 7f1eaa73b640 1 -- 172.21.15.6:0/2167964985 learned_addr learned my addr 172.21.15.6:0/2167964985 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.291+0000 7f1ea9f3a640 1 --2- 172.21.15.6:0/2167964985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea415bd40 0x7f1ea40d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea8f38640 1 -- 172.21.15.6:0/2167964985 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1952734055 0 0) 0x7f1ea40cc870 con 0x7f1ea40cfea0 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea8f38640 1 -- 172.21.15.6:0/2167964985 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e8c003520 con 0x7f1ea40cfea0 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea8f38640 1 -- 172.21.15.6:0/2167964985 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309730607 0 0) 0x7f1ea40d0500 con 0x7f1ea40d1d90 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea8f38640 1 -- 172.21.15.6:0/2167964985 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ea40cc870 con 0x7f1ea40d1d90 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea9f3a640 1 -- 172.21.15.6:0/2167964985 >> v1:172.21.15.92:6789/0 conn(0x7f1ea40d1d90 legacy=0x7f1ea40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea9f3a640 1 -- 172.21.15.6:0/2167964985 >> v1:172.21.15.6:6789/0 conn(0x7f1ea40cfea0 legacy=0x7f1ea40cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea9f3a640 1 -- 172.21.15.6:0/2167964985 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea40ccb10 con 0x7f1ea415bd40 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.292+0000 7f1ea9f3a640 1 --2- 172.21.15.6:0/2167964985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea415bd40 0x7f1ea40d1850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009dc0 tx=0x7f1e9402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6eb78953ecbab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1ea8f38640 1 -- 172.21.15.6:0/2167964985 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94039070 con 0x7f1ea415bd40 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1ea8f38640 1 -- 172.21.15.6:0/2167964985 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9402ccd0 con 0x7f1ea415bd40 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1ea8f38640 1 -- 172.21.15.6:0/2167964985 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94034640 con 0x7f1ea415bd40 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1eac19e640 1 -- 172.21.15.6:0/2167964985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea415bd40 msgr2=0x7f1ea40d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1eac19e640 1 --2- 172.21.15.6:0/2167964985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea415bd40 0x7f1ea40d1850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009dc0 tx=0x7f1e9402bf20 comp rx=0 tx=0).stop 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1eac19e640 1 -- 172.21.15.6:0/2167964985 shutdown_connections 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1eac19e640 1 --2- 172.21.15.6:0/2167964985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea415bd40 0x7f1ea40d1850 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1eac19e640 1 -- 172.21.15.6:0/2167964985 >> 172.21.15.6:0/2167964985 conn(0x7f1ea409ae20 msgr2=0x7f1ea408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1eac19e640 1 -- 172.21.15.6:0/2167964985 shutdown_connections 2024-03-18T14:42:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.293+0000 7f1eac19e640 1 -- 172.21.15.6:0/2167964985 wait complete. 2024-03-18T14:42:34.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.294+0000 7f1eac19e640 1 Processor -- start 2024-03-18T14:42:34.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.294+0000 7f1eac19e640 1 -- start start 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1eac19e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ea40cfea0 0x7f1ea41f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1eac19e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea40d1d90 0x7f1ea4158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1eac19e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea415bd40 0x7f1ea40ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1eac19e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea40d0c40 con 0x7f1ea40cfea0 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1eac19e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea40d0ac0 con 0x7f1ea415bd40 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1eac19e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea415c120 con 0x7f1ea40d1d90 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9f3a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ea40cfea0 0x7f1ea41f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9f3a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ea40cfea0 0x7f1ea41f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36776/0 (socket says 172.21.15.6:36776) 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9f3a640 1 -- 172.21.15.6:0/3863817885 learned_addr learned my addr 172.21.15.6:0/3863817885 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1eaa73b640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea415bd40 0x7f1ea40ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9739640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea40d1d90 0x7f1ea4158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9f3a640 1 -- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea40d1d90 msgr2=0x7f1ea4158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9f3a640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea40d1d90 0x7f1ea4158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9f3a640 1 -- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea415bd40 msgr2=0x7f1ea40ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.295+0000 7f1ea9f3a640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea415bd40 0x7f1ea40ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1ea9f3a640 1 -- 172.21.15.6:0/3863817885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea40ae960 con 0x7f1ea40cfea0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1eaa73b640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea415bd40 0x7f1ea40ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1ea9739640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea40d1d90 0x7f1ea4158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1ea9f3a640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ea40cfea0 0x7f1ea41f72f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009ee0 tx=0x7f1e94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1ea27fc640 1 -- 172.21.15.6:0/3863817885 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94039050 con 0x7f1ea40cfea0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1ea27fc640 1 -- 172.21.15.6:0/3863817885 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e94045070 con 0x7f1ea40cfea0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea40aec40 con 0x7f1ea40cfea0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1ea27fc640 1 -- 172.21.15.6:0/3863817885 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94009450 con 0x7f1ea40cfea0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.296+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea41586b0 con 0x7f1ea40cfea0 2024-03-18T14:42:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.297+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea409fa90 con 0x7f1ea40cfea0 2024-03-18T14:42:34.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.297+0000 7f1ea27fc640 1 -- 172.21.15.6:0/3863817885 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1e9403e830 con 0x7f1ea40cfea0 2024-03-18T14:42:34.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.298+0000 7f1ea27fc640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1e9006bbd0 0x7f1e9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.298+0000 7f1ea27fc640 1 -- 172.21.15.6:0/3863817885 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1e940adb90 con 0x7f1ea40cfea0 2024-03-18T14:42:34.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.298+0000 7f1ea9739640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1e9006bbd0 0x7f1e9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.298+0000 7f1ea9739640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1e9006bbd0 0x7f1e9006e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ea415f550 tx=0x7f1e98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.302+0000 7f1ea27fc640 1 -- 172.21.15.6:0/3863817885 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1e94032e20 con 0x7f1ea40cfea0 2024-03-18T14:42:34.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.483+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ea404f620 con 0x7f1ea40cfea0 2024-03-18T14:42:34.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.485+0000 7f1ea27fc640 1 -- 172.21.15.6:0/3863817885 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7f1e9407c460 con 0x7f1ea40cfea0 2024-03-18T14:42:34.487 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:34.488 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.486+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1e9006bbd0 msgr2=0x7f1e9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.486+0000 7f1eac19e640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1e9006bbd0 0x7f1e9006e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ea415f550 tx=0x7f1e98009040 comp rx=0 tx=0).stop 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.486+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ea40cfea0 msgr2=0x7f1ea41f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.486+0000 7f1eac19e640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ea40cfea0 0x7f1ea41f72f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009ee0 tx=0x7f1e94004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.486+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 shutdown_connections 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.486+0000 7f1eac19e640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1e9006bbd0 0x7f1e9006e090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.487+0000 7f1eac19e640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea415bd40 0x7f1ea40ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.487+0000 7f1eac19e640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea40d1d90 0x7f1ea4158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.487+0000 7f1eac19e640 1 --2- 172.21.15.6:0/3863817885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ea40cfea0 0x7f1ea41f72f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.487+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 >> 172.21.15.6:0/3863817885 conn(0x7f1ea409ae20 msgr2=0x7f1ea415b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.487+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 shutdown_connections 2024-03-18T14:42:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.487+0000 7f1eac19e640 1 -- 172.21.15.6:0/3863817885 wait complete. 2024-03-18T14:42:34.500 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:34.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.603+0000 7f37fcd9d640 1 Processor -- start 2024-03-18T14:42:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.603+0000 7f37fcd9d640 1 -- start start 2024-03-18T14:42:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37fcd9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f815dd50 0x7f37f80b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37fcd9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37f815e130 0x7f37f80ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37fcd9d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f8161bc0 con 0x7f37f80b55c0 2024-03-18T14:42:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37fcd9d640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f37f8161ea0 con 0x7f37f815e130 2024-03-18T14:42:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37fcd9d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f37f8162180 con 0x7f37f815dd50 2024-03-18T14:42:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f6ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37f815e130 0x7f37f80ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f6ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37f815e130 0x7f37f80ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36794/0 (socket says 172.21.15.6:36794) 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f6ffd640 1 -- 172.21.15.6:0/209671131 learned_addr learned my addr 172.21.15.6:0/209671131 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f77fe640 1 --2- 172.21.15.6:0/209671131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f815dd50 0x7f37f80b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f6ffd640 1 -- 172.21.15.6:0/209671131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f815dd50 msgr2=0x7f37f80b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f6ffd640 1 --2- 172.21.15.6:0/209671131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f815dd50 0x7f37f80b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f6ffd640 1 -- 172.21.15.6:0/209671131 >> v1:172.21.15.92:6789/0 conn(0x7f37f80b55c0 legacy=0x7f37f80b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f6ffd640 1 -- 172.21.15.6:0/209671131 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f8162460 con 0x7f37f815e130 2024-03-18T14:42:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.604+0000 7f37f67fc640 1 -- 172.21.15.6:0/209671131 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028981027 0 0) 0x7f37f8161bc0 con 0x7f37f80b55c0 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.605+0000 7f37f6ffd640 1 --2- 172.21.15.6:0/209671131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f815e130 0x7f37f80ce3a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009870 tx=0x7f37ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39f12ee72aedd97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.605+0000 7f37f67fc640 1 -- 172.21.15.6:0/209671131 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec034070 con 0x7f37f815e130 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.605+0000 7f37f67fc640 1 -- 172.21.15.6:0/209671131 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ec030840 con 0x7f37f815e130 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.605+0000 7f37f67fc640 1 -- 172.21.15.6:0/209671131 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec030c10 con 0x7f37f815e130 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 -- 172.21.15.6:0/209671131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f815e130 msgr2=0x7f37f80ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/209671131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f815e130 0x7f37f80ce3a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009870 tx=0x7f37ec02b980 comp rx=0 tx=0).stop 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 -- 172.21.15.6:0/209671131 shutdown_connections 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/209671131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f815e130 0x7f37f80ce3a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/209671131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f815dd50 0x7f37f80b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 -- 172.21.15.6:0/209671131 >> 172.21.15.6:0/209671131 conn(0x7f37f80d05c0 msgr2=0x7f37f80d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 -- 172.21.15.6:0/209671131 shutdown_connections 2024-03-18T14:42:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.606+0000 7f37fcd9d640 1 -- 172.21.15.6:0/209671131 wait complete. 2024-03-18T14:42:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.607+0000 7f37fcd9d640 1 Processor -- start 2024-03-18T14:42:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.607+0000 7f37fcd9d640 1 -- start start 2024-03-18T14:42:34.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37fcd9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f80b55c0 0x7f37f81f7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37fcd9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f815dd50 0x7f37f814a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37fcd9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f8149630 0x7f37f8149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37fcd9d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f37f80d1fd0 con 0x7f37f8149630 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37fcd9d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f37f80d1e50 con 0x7f37f815dd50 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37fcd9d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f37f80d2150 con 0x7f37f80b55c0 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f8149630 0x7f37f8149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f8149630 0x7f37f8149ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36808/0 (socket says 172.21.15.6:36808) 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f77fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f80b55c0 0x7f37f81f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 -- 172.21.15.6:0/1489933007 learned_addr learned my addr 172.21.15.6:0/1489933007 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f6ffd640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f815dd50 0x7f37f814a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 -- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f80b55c0 msgr2=0x7f37f81f7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f80b55c0 0x7f37f81f7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 -- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f815dd50 msgr2=0x7f37f814a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f815dd50 0x7f37f814a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.608+0000 7f37f7fff640 1 -- 172.21.15.6:0/1489933007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f814a020 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.609+0000 7f37f7fff640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f8149630 0x7f37f8149ae0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f37f000c760 tx=0x7f37f000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.609+0000 7f37dffff640 1 -- 172.21.15.6:0/1489933007 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0003da0 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.609+0000 7f37dffff640 1 -- 172.21.15.6:0/1489933007 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f0005070 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.609+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f80c8950 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.609+0000 7f37dffff640 1 -- 172.21.15.6:0/1489933007 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0005460 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.609+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f80c8f10 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.610+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f804f540 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.610+0000 7f37dffff640 1 -- 172.21.15.6:0/1489933007 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f37f0005600 con 0x7f37f8149630 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.610+0000 7f37dffff640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37e406bc00 0x7f37e406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.611+0000 7f37dffff640 1 -- 172.21.15.6:0/1489933007 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f37f008ab40 con 0x7f37f8149630 2024-03-18T14:42:34.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.611+0000 7f37f77fe640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37e406bc00 0x7f37e406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.611+0000 7f37f77fe640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37e406bc00 0x7f37e406e0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009b40 tx=0x7f37e8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.615+0000 7f37dffff640 1 -- 172.21.15.6:0/1489933007 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f37f0058c70 con 0x7f37f8149630 2024-03-18T14:42:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.794+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f37f80d1850 con 0x7f37f8149630 2024-03-18T14:42:34.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.795+0000 7f37dffff640 1 -- 172.21.15.6:0/1489933007 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f37f005cd10 con 0x7f37f8149630 2024-03-18T14:42:34.798 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37e406bc00 msgr2=0x7f37e406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37e406bc00 0x7f37e406e0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009b40 tx=0x7f37e8009190 comp rx=0 tx=0).stop 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f8149630 msgr2=0x7f37f8149ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f8149630 0x7f37f8149ae0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f37f000c760 tx=0x7f37f000cc20 comp rx=0 tx=0).stop 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 shutdown_connections 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f37e406bc00 0x7f37e406e0c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37f8149630 0x7f37f8149ae0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f815dd50 0x7f37f814a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 --2- 172.21.15.6:0/1489933007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37f80b55c0 0x7f37f81f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 >> 172.21.15.6:0/1489933007 conn(0x7f37f80d05c0 msgr2=0x7f37f80d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 shutdown_connections 2024-03-18T14:42:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.797+0000 7f37fcd9d640 1 -- 172.21.15.6:0/1489933007 wait complete. 2024-03-18T14:42:34.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:34.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.912+0000 7f71cb219640 1 Processor -- start 2024-03-18T14:42:34.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.912+0000 7f71cb219640 1 -- start start 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71cb219640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f71c409a350 0x7f71c40cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71cb219640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71c40d0300 con 0x7f71c40d13a0 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71cb219640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71c40d05e0 con 0x7f71c415e150 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71cb219640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f71c415e530 con 0x7f71c409a350 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71c3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f71c40d13a0 0x7f71c4099c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49936/0 (socket says 172.21.15.6:49936) 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71c3fff640 1 -- 172.21.15.6:0/3649884239 learned_addr learned my addr 172.21.15.6:0/3649884239 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71c97b6640 1 --2- 172.21.15.6:0/3649884239 >> v2:172.21.15.92:3300/0 conn(0x7f71c409a350 0x7f71c40cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71c37fe640 1 -- 172.21.15.6:0/3649884239 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1117746210 0 0) 0x7f71c40d0300 con 0x7f71c40d13a0 2024-03-18T14:42:34.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71c37fe640 1 -- 172.21.15.6:0/3649884239 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71ac003520 con 0x7f71c40d13a0 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71c37fe640 1 -- 172.21.15.6:0/3649884239 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368017284 0 0) 0x7f71c40d05e0 con 0x7f71c415e150 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.913+0000 7f71c37fe640 1 -- 172.21.15.6:0/3649884239 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c40d0300 con 0x7f71c415e150 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.914+0000 7f71c97b6640 1 -- 172.21.15.6:0/3649884239 >> v1:172.21.15.92:6789/0 conn(0x7f71c415e150 legacy=0x7f71c406d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.914+0000 7f71c97b6640 1 -- 172.21.15.6:0/3649884239 >> v1:172.21.15.6:6789/0 conn(0x7f71c40d13a0 legacy=0x7f71c4099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.914+0000 7f71c97b6640 1 -- 172.21.15.6:0/3649884239 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c40d08c0 con 0x7f71c409a350 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.914+0000 7f71c97b6640 1 --2- 172.21.15.6:0/3649884239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c409a350 0x7f71c40cc1b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009d90 tx=0x7f71bc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f0a170f62b85cf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.914+0000 7f71c37fe640 1 -- 172.21.15.6:0/3649884239 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc039070 con 0x7f71c409a350 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71c37fe640 1 -- 172.21.15.6:0/3649884239 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71bc02cc60 con 0x7f71c409a350 2024-03-18T14:42:34.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71c37fe640 1 -- 172.21.15.6:0/3649884239 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc034660 con 0x7f71c409a350 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71cb219640 1 -- 172.21.15.6:0/3649884239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c409a350 msgr2=0x7f71c40cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71cb219640 1 --2- 172.21.15.6:0/3649884239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c409a350 0x7f71c40cc1b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009d90 tx=0x7f71bc02beb0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71cb219640 1 -- 172.21.15.6:0/3649884239 shutdown_connections 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71cb219640 1 --2- 172.21.15.6:0/3649884239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c409a350 0x7f71c40cc1b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71cb219640 1 -- 172.21.15.6:0/3649884239 >> 172.21.15.6:0/3649884239 conn(0x7f71c40b3650 msgr2=0x7f71c40b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71cb219640 1 -- 172.21.15.6:0/3649884239 shutdown_connections 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.915+0000 7f71cb219640 1 -- 172.21.15.6:0/3649884239 wait complete. 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.916+0000 7f71cb219640 1 Processor -- start 2024-03-18T14:42:34.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.916+0000 7f71cb219640 1 -- start start 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71cb219640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c409a350 0x7f71c40c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71cb219640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c40d13a0 0x7f71c40b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71cb219640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71c415e150 0x7f71c415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71cb219640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f71c406c650 con 0x7f71c409a350 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71cb219640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f71c40d0e80 con 0x7f71c40d13a0 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71cb219640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f71c406c7d0 con 0x7f71c415e150 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71c8fb5640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c409a350 0x7f71c40c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71c8fb5640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c409a350 0x7f71c40c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36822/0 (socket says 172.21.15.6:36822) 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71c8fb5640 1 -- 172.21.15.6:0/2779882169 learned_addr learned my addr 172.21.15.6:0/2779882169 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71c3fff640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c40d13a0 0x7f71c40b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.917+0000 7f71c97b6640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71c415e150 0x7f71c415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c8fb5640 1 -- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71c415e150 msgr2=0x7f71c415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c8fb5640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71c415e150 0x7f71c415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c8fb5640 1 -- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c40d13a0 msgr2=0x7f71c40b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:34.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c8fb5640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c40d13a0 0x7f71c40b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:34.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c8fb5640 1 -- 172.21.15.6:0/2779882169 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c40c79b0 con 0x7f71c409a350 2024-03-18T14:42:34.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c8fb5640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c409a350 0x7f71c40c70b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f71b400c460 tx=0x7f71b400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c17fa640 1 -- 172.21.15.6:0/2779882169 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4017070 con 0x7f71c409a350 2024-03-18T14:42:34.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c17fa640 1 -- 172.21.15.6:0/2779882169 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71b40052f0 con 0x7f71c409a350 2024-03-18T14:42:34.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c40c7c90 con 0x7f71c409a350 2024-03-18T14:42:34.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71c17fa640 1 -- 172.21.15.6:0/2779882169 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4003830 con 0x7f71c409a350 2024-03-18T14:42:34.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.918+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c415f940 con 0x7f71c409a350 2024-03-18T14:42:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.919+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71c40751b0 con 0x7f71c409a350 2024-03-18T14:42:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.920+0000 7f71c17fa640 1 -- 172.21.15.6:0/2779882169 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f71b401a070 con 0x7f71c409a350 2024-03-18T14:42:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.920+0000 7f71c17fa640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f71b006bbd0 0x7f71b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.920+0000 7f71c17fa640 1 -- 172.21.15.6:0/2779882169 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f71b408a220 con 0x7f71c409a350 2024-03-18T14:42:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.920+0000 7f71c3fff640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f71b006bbd0 0x7f71b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.921+0000 7f71c3fff640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f71b006bbd0 0x7f71b006e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f71c406ccf0 tx=0x7f71b8007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:34.924+0000 7f71c17fa640 1 -- 172.21.15.6:0/2779882169 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f71b4058240 con 0x7f71c409a350 2024-03-18T14:42:35.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.105+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f71c40cf7d0 con 0x7f71c409a350 2024-03-18T14:42:35.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.105+0000 7f71c17fa640 1 -- 172.21.15.6:0/2779882169 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f71b405c2e0 con 0x7f71c409a350 2024-03-18T14:42:35.108 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:35.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f71b006bbd0 msgr2=0x7f71b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f71b006bbd0 0x7f71b006e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f71c406ccf0 tx=0x7f71b8007d30 comp rx=0 tx=0).stop 2024-03-18T14:42:35.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c409a350 msgr2=0x7f71c40c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c409a350 0x7f71c40c70b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f71b400c460 tx=0x7f71b400c920 comp rx=0 tx=0).stop 2024-03-18T14:42:35.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 shutdown_connections 2024-03-18T14:42:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f71b006bbd0 0x7f71b006e090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71c415e150 0x7f71c415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71c40d13a0 0x7f71c40b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 --2- 172.21.15.6:0/2779882169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71c409a350 0x7f71c40c70b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 >> 172.21.15.6:0/2779882169 conn(0x7f71c40b3650 msgr2=0x7f71c40b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.107+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 shutdown_connections 2024-03-18T14:42:35.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.108+0000 7f71cb219640 1 -- 172.21.15.6:0/2779882169 wait complete. 2024-03-18T14:42:35.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:35.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:42:35.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:42:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.223+0000 7f1e0a3df640 1 Processor -- start 2024-03-18T14:42:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.223+0000 7f1e0a3df640 1 -- start start 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0a3df640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1e0409a350 0x7f1e040cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0a3df640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e040d0300 con 0x7f1e040d13a0 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0a3df640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e040d05e0 con 0x7f1e0415e150 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0a3df640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1e0415e530 con 0x7f1e0409a350 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e037fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1e040d13a0 0x7f1e04099c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49946/0 (socket says 172.21.15.6:49946) 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e037fe640 1 -- 172.21.15.6:0/4016115694 learned_addr learned my addr 172.21.15.6:0/4016115694 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0897c640 1 --2- 172.21.15.6:0/4016115694 >> v2:172.21.15.92:3300/0 conn(0x7f1e0409a350 0x7f1e040cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e02ffd640 1 -- 172.21.15.6:0/4016115694 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2230828539 0 0) 0x7f1e040d0300 con 0x7f1e040d13a0 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e02ffd640 1 -- 172.21.15.6:0/4016115694 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dec003520 con 0x7f1e040d13a0 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0897c640 1 -- 172.21.15.6:0/4016115694 >> v1:172.21.15.92:6789/0 conn(0x7f1e0415e150 legacy=0x7f1e0406d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0897c640 1 -- 172.21.15.6:0/4016115694 >> v1:172.21.15.6:6789/0 conn(0x7f1e040d13a0 legacy=0x7f1e04099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.224+0000 7f1e0897c640 1 -- 172.21.15.6:0/4016115694 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e040d08c0 con 0x7f1e0409a350 2024-03-18T14:42:35.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.225+0000 7f1e0897c640 1 --2- 172.21.15.6:0/4016115694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0409a350 0x7f1e040cc1b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009d90 tx=0x7f1dfc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66a0cadc724b2ac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.225+0000 7f1e02ffd640 1 -- 172.21.15.6:0/4016115694 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc039070 con 0x7f1e0409a350 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.225+0000 7f1e02ffd640 1 -- 172.21.15.6:0/4016115694 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dfc02cc60 con 0x7f1e0409a350 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.225+0000 7f1e02ffd640 1 -- 172.21.15.6:0/4016115694 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc034640 con 0x7f1e0409a350 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.226+0000 7f1e0a3df640 1 -- 172.21.15.6:0/4016115694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0409a350 msgr2=0x7f1e040cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.226+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/4016115694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0409a350 0x7f1e040cc1b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009d90 tx=0x7f1dfc02beb0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.226+0000 7f1e0a3df640 1 -- 172.21.15.6:0/4016115694 shutdown_connections 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.226+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/4016115694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0409a350 0x7f1e040cc1b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.226+0000 7f1e0a3df640 1 -- 172.21.15.6:0/4016115694 >> 172.21.15.6:0/4016115694 conn(0x7f1e040b3650 msgr2=0x7f1e040b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:35.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.226+0000 7f1e0a3df640 1 -- 172.21.15.6:0/4016115694 shutdown_connections 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.226+0000 7f1e0a3df640 1 -- 172.21.15.6:0/4016115694 wait complete. 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.227+0000 7f1e0a3df640 1 Processor -- start 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.227+0000 7f1e0a3df640 1 -- start start 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0a3df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e0409a350 0x7f1e0415c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0a3df640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e040d13a0 0x7f1e0415c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0a3df640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e0415e150 0x7f1e040c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0a3df640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0406c650 con 0x7f1e0415e150 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0a3df640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e040d0e80 con 0x7f1e040d13a0 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0a3df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1e0406c7d0 con 0x7f1e0409a350 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e0415e150 0x7f1e040c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e0415e150 0x7f1e040c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36830/0 (socket says 172.21.15.6:36830) 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 -- 172.21.15.6:0/2486568861 learned_addr learned my addr 172.21.15.6:0/2486568861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e037fe640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e040d13a0 0x7f1e0415c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e03fff640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e0409a350 0x7f1e0415c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 -- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e0409a350 msgr2=0x7f1e0415c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e0409a350 0x7f1e0415c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 -- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e040d13a0 msgr2=0x7f1e0415c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e040d13a0 0x7f1e0415c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.228+0000 7f1e0897c640 1 -- 172.21.15.6:0/2486568861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e041616f0 con 0x7f1e0415e150 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.229+0000 7f1e0897c640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e0415e150 0x7f1e040c5690 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009eb0 tx=0x7f1dfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.229+0000 7f1e00ff9640 1 -- 172.21.15.6:0/2486568861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc042030 con 0x7f1e0415e150 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.229+0000 7f1e00ff9640 1 -- 172.21.15.6:0/2486568861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dfc03a040 con 0x7f1e0415e150 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.229+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e040c7db0 con 0x7f1e0415e150 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.229+0000 7f1e00ff9640 1 -- 172.21.15.6:0/2486568861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc039070 con 0x7f1e0415e150 2024-03-18T14:42:35.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.229+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e040c8370 con 0x7f1e0415e150 2024-03-18T14:42:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.230+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e040751b0 con 0x7f1e0415e150 2024-03-18T14:42:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.230+0000 7f1e00ff9640 1 -- 172.21.15.6:0/2486568861 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1dfc045070 con 0x7f1e0415e150 2024-03-18T14:42:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.231+0000 7f1e00ff9640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1df006bc00 0x7f1df006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.231+0000 7f1e00ff9640 1 -- 172.21.15.6:0/2486568861 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1dfc0acc70 con 0x7f1e0415e150 2024-03-18T14:42:35.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.231+0000 7f1e03fff640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1df006bc00 0x7f1df006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.231+0000 7f1e03fff640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1df006bc00 0x7f1df006e0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1df80016e0 tx=0x7f1df8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.235+0000 7f1e00ff9640 1 -- 172.21.15.6:0/2486568861 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1dfc04c070 con 0x7f1e0415e150 2024-03-18T14:42:35.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.416+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1e040b2910 con 0x7f1e0415e150 2024-03-18T14:42:35.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.417+0000 7f1e00ff9640 1 -- 172.21.15.6:0/2486568861 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f1dfc07ad10 con 0x7f1e0415e150 2024-03-18T14:42:35.420 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:35.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1df006bc00 msgr2=0x7f1df006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1df006bc00 0x7f1df006e0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1df80016e0 tx=0x7f1df8009040 comp rx=0 tx=0).stop 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e0415e150 msgr2=0x7f1e040c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e0415e150 0x7f1e040c5690 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009eb0 tx=0x7f1dfc004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 shutdown_connections 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1df006bc00 0x7f1df006e0c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e0415e150 0x7f1e040c5690 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e040d13a0 0x7f1e0415c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 --2- 172.21.15.6:0/2486568861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e0409a350 0x7f1e0415c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 >> 172.21.15.6:0/2486568861 conn(0x7f1e040b3650 msgr2=0x7f1e04099c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 shutdown_connections 2024-03-18T14:42:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.419+0000 7f1e0a3df640 1 -- 172.21.15.6:0/2486568861 wait complete. 2024-03-18T14:42:35.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:35.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.534+0000 7f0d47d7e640 1 Processor -- start 2024-03-18T14:42:35.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.534+0000 7f0d47d7e640 1 -- start start 2024-03-18T14:42:35.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d47d7e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d400b5e50 0x7f0d400b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d47d7e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d4015e3b0 con 0x7f0d400d2e50 2024-03-18T14:42:35.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d47d7e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d400b8470 con 0x7f0d4015dfd0 2024-03-18T14:42:35.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d47d7e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0d400b8750 con 0x7f0d400b5e50 2024-03-18T14:42:35.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d4631b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d400b5e50 0x7f0d400b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d45319640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0d400d2e50 0x7f0d400b5740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49960/0 (socket says 172.21.15.6:49960) 2024-03-18T14:42:35.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d45319640 1 -- 172.21.15.6:0/1053750 learned_addr learned my addr 172.21.15.6:0/1053750 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:35.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d4631b640 1 -- 172.21.15.6:0/1053750 >> v1:172.21.15.92:6789/0 conn(0x7f0d4015dfd0 legacy=0x7f0d400b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d4631b640 1 -- 172.21.15.6:0/1053750 >> v1:172.21.15.6:6789/0 conn(0x7f0d400d2e50 legacy=0x7f0d400b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.535+0000 7f0d4631b640 1 -- 172.21.15.6:0/1053750 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d400b8a30 con 0x7f0d400b5e50 2024-03-18T14:42:35.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.536+0000 7f0d4631b640 1 --2- 172.21.15.6:0/1053750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d400b7e00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009d90 tx=0x7f0d3802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f399a87a0ba4ed46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.536+0000 7f0d44b18640 1 -- 172.21.15.6:0/1053750 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38039070 con 0x7f0d400b5e50 2024-03-18T14:42:35.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.536+0000 7f0d44b18640 1 -- 172.21.15.6:0/1053750 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3802cc60 con 0x7f0d400b5e50 2024-03-18T14:42:35.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.536+0000 7f0d44b18640 1 -- 172.21.15.6:0/1053750 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38034660 con 0x7f0d400b5e50 2024-03-18T14:42:35.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.536+0000 7f0d47d7e640 1 -- 172.21.15.6:0/1053750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 msgr2=0x7f0d400b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.536+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/1053750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d400b7e00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009d90 tx=0x7f0d3802beb0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.537+0000 7f0d47d7e640 1 -- 172.21.15.6:0/1053750 shutdown_connections 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.537+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/1053750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d400b7e00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.537+0000 7f0d47d7e640 1 -- 172.21.15.6:0/1053750 >> 172.21.15.6:0/1053750 conn(0x7f0d40072d10 msgr2=0x7f0d40073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.537+0000 7f0d47d7e640 1 -- 172.21.15.6:0/1053750 shutdown_connections 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.537+0000 7f0d47d7e640 1 -- 172.21.15.6:0/1053750 wait complete. 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.538+0000 7f0d47d7e640 1 Processor -- start 2024-03-18T14:42:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.538+0000 7f0d47d7e640 1 -- start start 2024-03-18T14:42:35.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.538+0000 7f0d47d7e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d4015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.538+0000 7f0d47d7e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d400d2e50 0x7f0d4015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d47d7e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4015dfd0 0x7f0d400c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d47d7e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0d40168c60 con 0x7f0d400b5e50 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d47d7e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0d40168ae0 con 0x7f0d4015dfd0 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d47d7e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0d40168de0 con 0x7f0d400d2e50 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d4015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d4015a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36850/0 (socket says 172.21.15.6:36850) 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 -- 172.21.15.6:0/2856660935 learned_addr learned my addr 172.21.15.6:0/2856660935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45319640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d400d2e50 0x7f0d4015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d4631b640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4015dfd0 0x7f0d400c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 -- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d400d2e50 msgr2=0x7f0d4015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d400d2e50 0x7f0d4015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 -- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4015dfd0 msgr2=0x7f0d400c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4015dfd0 0x7f0d400c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 -- 172.21.15.6:0/2856660935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4015c870 con 0x7f0d400b5e50 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d4631b640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4015dfd0 0x7f0d400c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45319640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d400d2e50 0x7f0d4015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d45b1a640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d4015a0b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c3f0 tx=0x7f0d3c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d367fc640 1 -- 172.21.15.6:0/2856660935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0052d0 con 0x7f0d400b5e50 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d367fc640 1 -- 172.21.15.6:0/2856660935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3c005490 con 0x7f0d400b5e50 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4015a740 con 0x7f0d400b5e50 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d367fc640 1 -- 172.21.15.6:0/2856660935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c007260 con 0x7f0d400b5e50 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.539+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4015ad00 con 0x7f0d400b5e50 2024-03-18T14:42:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.540+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d400751b0 con 0x7f0d400b5e50 2024-03-18T14:42:35.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.541+0000 7f0d367fc640 1 -- 172.21.15.6:0/2856660935 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0d3c007400 con 0x7f0d400b5e50 2024-03-18T14:42:35.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.541+0000 7f0d367fc640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0d2406bbd0 0x7f0d2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.541+0000 7f0d367fc640 1 -- 172.21.15.6:0/2856660935 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0d3c08a2c0 con 0x7f0d400b5e50 2024-03-18T14:42:35.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.541+0000 7f0d45319640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0d2406bbd0 0x7f0d2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.542+0000 7f0d45319640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0d2406bbd0 0x7f0d2406e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0d40168a40 tx=0x7f0d30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.546+0000 7f0d367fc640 1 -- 172.21.15.6:0/2856660935 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0d3c058a40 con 0x7f0d400b5e50 2024-03-18T14:42:35.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.731+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0d401567a0 con 0x7f0d400b5e50 2024-03-18T14:42:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.731+0000 7f0d367fc640 1 -- 172.21.15.6:0/2856660935 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f0d3c058340 con 0x7f0d400b5e50 2024-03-18T14:42:35.734 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0d2406bbd0 msgr2=0x7f0d2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0d2406bbd0 0x7f0d2406e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0d40168a40 tx=0x7f0d30009040 comp rx=0 tx=0).stop 2024-03-18T14:42:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 msgr2=0x7f0d4015a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d4015a0b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c3f0 tx=0x7f0d3c00c8b0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 shutdown_connections 2024-03-18T14:42:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0d2406bbd0 0x7f0d2406e090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4015dfd0 0x7f0d400c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d400d2e50 0x7f0d4015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 --2- 172.21.15.6:0/2856660935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d400b5e50 0x7f0d4015a0b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 >> 172.21.15.6:0/2856660935 conn(0x7f0d40072d10 msgr2=0x7f0d40155a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 shutdown_connections 2024-03-18T14:42:35.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.733+0000 7f0d47d7e640 1 -- 172.21.15.6:0/2856660935 wait complete. 2024-03-18T14:42:35.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.847+0000 7f786dcc6640 1 Processor -- start 2024-03-18T14:42:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.847+0000 7f786dcc6640 1 -- start start 2024-03-18T14:42:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f786dcc6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f786806caf0 0x7f78681624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f786dcc6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f786806f9a0 0x7f78680acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f786dcc6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78680722c0 con 0x7f786815e010 2024-03-18T14:42:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f786dcc6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f78681629f0 con 0x7f786806f9a0 2024-03-18T14:42:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f786dcc6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f786815e3f0 con 0x7f786806caf0 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f7867fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f786806f9a0 0x7f78680acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f78677fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f786815e010 0x7f786806c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49968/0 (socket says 172.21.15.6:49968) 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f78677fe640 1 -- 172.21.15.6:0/29326203 learned_addr learned my addr 172.21.15.6:0/29326203 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f7866ffd640 1 --2- 172.21.15.6:0/29326203 >> v2:172.21.15.92:3300/0 conn(0x7f786806caf0 0x7f78681624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f7867fff640 1 -- 172.21.15.6:0/29326203 >> v2:172.21.15.92:3300/0 conn(0x7f786806caf0 msgr2=0x7f78681624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f7867fff640 1 --2- 172.21.15.6:0/29326203 >> v2:172.21.15.92:3300/0 conn(0x7f786806caf0 0x7f78681624b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f7867fff640 1 -- 172.21.15.6:0/29326203 >> v1:172.21.15.6:6789/0 conn(0x7f786815e010 legacy=0x7f786806c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f7867fff640 1 -- 172.21.15.6:0/29326203 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78680725a0 con 0x7f786806f9a0 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.848+0000 7f78667fc640 1 -- 172.21.15.6:0/29326203 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1885649654 0 0) 0x7f78680722c0 con 0x7f786815e010 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f7867fff640 1 --2- 172.21.15.6:0/29326203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806f9a0 0x7f78680acee0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7868161a00 tx=0x7f786002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a810fe2747144e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f78667fc640 1 -- 172.21.15.6:0/29326203 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860039070 con 0x7f786806f9a0 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f78667fc640 1 -- 172.21.15.6:0/29326203 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786002cd00 con 0x7f786806f9a0 2024-03-18T14:42:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f78667fc640 1 -- 172.21.15.6:0/29326203 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860034660 con 0x7f786806f9a0 2024-03-18T14:42:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f786dcc6640 1 -- 172.21.15.6:0/29326203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806f9a0 msgr2=0x7f78680acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f786dcc6640 1 --2- 172.21.15.6:0/29326203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806f9a0 0x7f78680acee0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7868161a00 tx=0x7f786002bf50 comp rx=0 tx=0).stop 2024-03-18T14:42:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f786dcc6640 1 -- 172.21.15.6:0/29326203 shutdown_connections 2024-03-18T14:42:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f786dcc6640 1 --2- 172.21.15.6:0/29326203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806f9a0 0x7f78680acee0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f786dcc6640 1 --2- 172.21.15.6:0/29326203 >> v2:172.21.15.92:3300/0 conn(0x7f786806caf0 0x7f78681624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f786dcc6640 1 -- 172.21.15.6:0/29326203 >> 172.21.15.6:0/29326203 conn(0x7f78680d0110 msgr2=0x7f78680d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.849+0000 7f786dcc6640 1 -- 172.21.15.6:0/29326203 shutdown_connections 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.850+0000 7f786dcc6640 1 -- 172.21.15.6:0/29326203 wait complete. 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 Processor -- start 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 -- start start 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806caf0 0x7f78680b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786806f9a0 0x7f78681601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f786815e010 0x7f7868151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7868072b60 con 0x7f786806caf0 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f786806e480 con 0x7f786806f9a0 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.851+0000 7f786dcc6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7868072e40 con 0x7f786815e010 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806caf0 0x7f78680b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806caf0 0x7f78680b4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36866/0 (socket says 172.21.15.6:36866) 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 -- 172.21.15.6:0/1144479980 learned_addr learned my addr 172.21.15.6:0/1144479980 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f7866ffd640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786806f9a0 0x7f78681601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f7867fff640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f786815e010 0x7f7868151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 -- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f786815e010 msgr2=0x7f7868151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f786815e010 0x7f7868151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 -- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786806f9a0 msgr2=0x7f78681601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786806f9a0 0x7f78681601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 -- 172.21.15.6:0/1144479980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786815e640 con 0x7f786806caf0 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f78677fe640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806caf0 0x7f78680b4f60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f785800c430 tx=0x7f785800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f784ffff640 1 -- 172.21.15.6:0/1144479980 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78580074e0 con 0x7f786806caf0 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f784ffff640 1 -- 172.21.15.6:0/1144479980 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78580047e0 con 0x7f786806caf0 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f784ffff640 1 -- 172.21.15.6:0/1144479980 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858004bd0 con 0x7f786806caf0 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786815e8b0 con 0x7f786806caf0 2024-03-18T14:42:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.852+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786815ee70 con 0x7f786806caf0 2024-03-18T14:42:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.853+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78680ae520 con 0x7f786806caf0 2024-03-18T14:42:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.854+0000 7f784ffff640 1 -- 172.21.15.6:0/1144479980 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7858011070 con 0x7f786806caf0 2024-03-18T14:42:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.854+0000 7f784ffff640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f785406bbd0 0x7f785406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.854+0000 7f784ffff640 1 -- 172.21.15.6:0/1144479980 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f785808a9d0 con 0x7f786806caf0 2024-03-18T14:42:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.854+0000 7f7866ffd640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f785406bbd0 0x7f785406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.855+0000 7f7866ffd640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f785406bbd0 0x7f785406e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f78680c3740 tx=0x7f785c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:35.858+0000 7f784ffff640 1 -- 172.21.15.6:0/1144479980 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f78680ae520 con 0x7f786806caf0 2024-03-18T14:42:36.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.054+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f78680b6090 con 0x7f786806caf0 2024-03-18T14:42:36.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.055+0000 7f784ffff640 1 -- 172.21.15.6:0/1144479980 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f78580589e0 con 0x7f786806caf0 2024-03-18T14:42:36.057 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.056+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f785406bbd0 msgr2=0x7f785406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f785406bbd0 0x7f785406e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f78680c3740 tx=0x7f785c009190 comp rx=0 tx=0).stop 2024-03-18T14:42:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806caf0 msgr2=0x7f78680b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806caf0 0x7f78680b4f60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f785800c430 tx=0x7f785800c8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 shutdown_connections 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f785406bbd0 0x7f785406e090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f786815e010 0x7f7868151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786806f9a0 0x7f78681601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 --2- 172.21.15.6:0/1144479980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f786806caf0 0x7f78680b4f60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 >> 172.21.15.6:0/1144479980 conn(0x7f78680d0110 msgr2=0x7f78680cf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 shutdown_connections 2024-03-18T14:42:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.057+0000 7f786dcc6640 1 -- 172.21.15.6:0/1144479980 wait complete. 2024-03-18T14:42:36.070 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:42:36.070 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:36.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:36.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.224+0000 7f4f0c0dc640 1 Processor -- start 2024-03-18T14:42:36.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.224+0000 7f4f0c0dc640 1 -- start start 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f0c0dc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f040b9650 0x7f4f040d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f0c0dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040d0bc0 0x7f4f040b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f0c0dc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f040d0ff0 con 0x7f4f0415e170 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f0c0dc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4f040b9ae0 con 0x7f4f040b9650 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f0c0dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4f040cef50 con 0x7f4f040d0bc0 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f09677640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f040b9650 0x7f4f040d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f09e78640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f0415e170 0x7f4f040d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49984/0 (socket says 172.21.15.6:49984) 2024-03-18T14:42:36.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f09e78640 1 -- 172.21.15.6:0/1343032044 learned_addr learned my addr 172.21.15.6:0/1343032044 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:36.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f0a679640 1 --2- 172.21.15.6:0/1343032044 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040d0bc0 0x7f4f040b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f09677640 1 -- 172.21.15.6:0/1343032044 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040d0bc0 msgr2=0x7f4f040b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:36.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f09677640 1 --2- 172.21.15.6:0/1343032044 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040d0bc0 0x7f4f040b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f09677640 1 -- 172.21.15.6:0/1343032044 >> v1:172.21.15.6:6789/0 conn(0x7f4f0415e170 legacy=0x7f4f040d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f09677640 1 -- 172.21.15.6:0/1343032044 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f040cf230 con 0x7f4f040b9650 2024-03-18T14:42:36.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.225+0000 7f4f08e76640 1 -- 172.21.15.6:0/1343032044 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028809721 0 0) 0x7f4f040d0ff0 con 0x7f4f0415e170 2024-03-18T14:42:36.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.226+0000 7f4f09677640 1 --2- 172.21.15.6:0/1343032044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040b9650 0x7f4f040d0680 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80097a0 tx=0x7f4ef802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbff97baa28a61ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.226+0000 7f4f08e76640 1 -- 172.21.15.6:0/1343032044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8035020 con 0x7f4f040b9650 2024-03-18T14:42:36.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.226+0000 7f4f08e76640 1 -- 172.21.15.6:0/1343032044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ef8031830 con 0x7f4f040b9650 2024-03-18T14:42:36.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.226+0000 7f4f08e76640 1 -- 172.21.15.6:0/1343032044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8031c00 con 0x7f4f040b9650 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1343032044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040b9650 msgr2=0x7f4f040d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1343032044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040b9650 0x7f4f040d0680 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80097a0 tx=0x7f4ef802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1343032044 shutdown_connections 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1343032044 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040d0bc0 0x7f4f040b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1343032044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040b9650 0x7f4f040d0680 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1343032044 >> 172.21.15.6:0/1343032044 conn(0x7f4f0406c900 msgr2=0x7f4f0406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1343032044 shutdown_connections 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.227+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1343032044 wait complete. 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.228+0000 7f4f0c0dc640 1 Processor -- start 2024-03-18T14:42:36.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.228+0000 7f4f0c0dc640 1 -- start start 2024-03-18T14:42:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f0c0dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040b9650 0x7f4f04158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f0c0dc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040d0bc0 0x7f4f04155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f0c0dc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f0415e170 0x7f4f04156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f0c0dc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f040d3a40 con 0x7f4f040d0bc0 2024-03-18T14:42:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f0c0dc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4f040d2e10 con 0x7f4f0415e170 2024-03-18T14:42:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f0c0dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4f040d2820 con 0x7f4f040b9650 2024-03-18T14:42:36.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040d0bc0 0x7f4f04155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040d0bc0 0x7f4f04155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36886/0 (socket says 172.21.15.6:36886) 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 -- 172.21.15.6:0/1651030152 learned_addr learned my addr 172.21.15.6:0/1651030152 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09e78640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040b9650 0x7f4f04158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f0a679640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f0415e170 0x7f4f04156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 -- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040b9650 msgr2=0x7f4f04158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040b9650 0x7f4f04158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 -- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f0415e170 msgr2=0x7f4f04156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f0415e170 0x7f4f04156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 -- 172.21.15.6:0/1651030152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f04158a80 con 0x7f4f040d0bc0 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.229+0000 7f4f09677640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040d0bc0 0x7f4f04155b00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4ef802bee0 tx=0x7f4ef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.230+0000 7f4f027fc640 1 -- 172.21.15.6:0/1651030152 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8035040 con 0x7f4f040d0bc0 2024-03-18T14:42:36.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.230+0000 7f4f027fc640 1 -- 172.21.15.6:0/1651030152 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ef803b920 con 0x7f4f040d0bc0 2024-03-18T14:42:36.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.230+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f04156670 con 0x7f4f040d0bc0 2024-03-18T14:42:36.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.230+0000 7f4f027fc640 1 -- 172.21.15.6:0/1651030152 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef803bd10 con 0x7f4f040d0bc0 2024-03-18T14:42:36.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.230+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f04156c30 con 0x7f4f040d0bc0 2024-03-18T14:42:36.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.231+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f0408afe0 con 0x7f4f040d0bc0 2024-03-18T14:42:36.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.231+0000 7f4f027fc640 1 -- 172.21.15.6:0/1651030152 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4ef8038020 con 0x7f4f040d0bc0 2024-03-18T14:42:36.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.231+0000 7f4f027fc640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ef006bbd0 0x7f4ef006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.231+0000 7f4f027fc640 1 -- 172.21.15.6:0/1651030152 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4ef80ace30 con 0x7f4f040d0bc0 2024-03-18T14:42:36.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.232+0000 7f4f09e78640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ef006bbd0 0x7f4ef006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.232+0000 7f4f09e78640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ef006bbd0 0x7f4ef006e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40012e0 tx=0x7f4ef4001630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.236+0000 7f4f027fc640 1 -- 172.21.15.6:0/1651030152 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4ef8031800 con 0x7f4f040d0bc0 2024-03-18T14:42:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.417+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f0415d700 con 0x7f4f040d0bc0 2024-03-18T14:42:36.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.418+0000 7f4f027fc640 1 -- 172.21.15.6:0/1651030152 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7f4ef807aee0 con 0x7f4f040d0bc0 2024-03-18T14:42:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:36.421 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ef006bbd0 msgr2=0x7f4ef006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ef006bbd0 0x7f4ef006e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40012e0 tx=0x7f4ef4001630 comp rx=0 tx=0).stop 2024-03-18T14:42:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040d0bc0 msgr2=0x7f4f04155b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040d0bc0 0x7f4f04155b00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4ef802bee0 tx=0x7f4ef80046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 shutdown_connections 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ef006bbd0 0x7f4ef006e090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f0415e170 0x7f4f04156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f040d0bc0 0x7f4f04155b00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 --2- 172.21.15.6:0/1651030152 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f040b9650 0x7f4f04158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.420+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 >> 172.21.15.6:0/1651030152 conn(0x7f4f0406c900 msgr2=0x7f4f040b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.421+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 shutdown_connections 2024-03-18T14:42:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.421+0000 7f4f0c0dc640 1 -- 172.21.15.6:0/1651030152 wait complete. 2024-03-18T14:42:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:36.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:36.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.537+0000 7fa3f4636640 1 Processor -- start 2024-03-18T14:42:36.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.537+0000 7fa3f4636640 1 -- start start 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f4636640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3ec0cf6a0 0x7fa3ec0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f4636640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec0cdb70 con 0x7fa3ec09a300 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f4636640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec0b1560 con 0x7fa3ec099f20 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f4636640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa3ec0b1800 con 0x7fa3ec0cf6a0 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f1bd1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3ec0cf6a0 0x7fa3ec0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f1bd1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3ec0cf6a0 0x7fa3ec0cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36890/0 (socket says 172.21.15.6:36890) 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f2bd3640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3ec09a300 0x7fa3ec0b29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49998/0 (socket says 172.21.15.6:49998) 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f1bd1640 1 -- 172.21.15.6:0/4101752759 learned_addr learned my addr 172.21.15.6:0/4101752759 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f1bd1640 1 -- 172.21.15.6:0/4101752759 >> v1:172.21.15.92:6789/0 conn(0x7fa3ec099f20 legacy=0x7fa3ec0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f1bd1640 1 -- 172.21.15.6:0/4101752759 >> v1:172.21.15.6:6789/0 conn(0x7fa3ec09a300 legacy=0x7fa3ec0b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f1bd1640 1 -- 172.21.15.6:0/4101752759 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec0b1ae0 con 0x7fa3ec0cf6a0 2024-03-18T14:42:36.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.538+0000 7fa3f13d0640 1 -- 172.21.15.6:0/4101752759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1867864107 0 0) 0x7fa3ec0cdb70 con 0x7fa3ec09a300 2024-03-18T14:42:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.539+0000 7fa3f1bd1640 1 --2- 172.21.15.6:0/4101752759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec0cf6a0 0x7fa3ec0cc790 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009870 tx=0x7fa3dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3678a2b07a2013b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.539+0000 7fa3f13d0640 1 -- 172.21.15.6:0/4101752759 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc035020 con 0x7fa3ec0cf6a0 2024-03-18T14:42:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.539+0000 7fa3f13d0640 1 -- 172.21.15.6:0/4101752759 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3dc031870 con 0x7fa3ec0cf6a0 2024-03-18T14:42:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.539+0000 7fa3f13d0640 1 -- 172.21.15.6:0/4101752759 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc031c40 con 0x7fa3ec0cf6a0 2024-03-18T14:42:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.539+0000 7fa3f4636640 1 -- 172.21.15.6:0/4101752759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec0cf6a0 msgr2=0x7fa3ec0cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.539+0000 7fa3f4636640 1 --2- 172.21.15.6:0/4101752759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec0cf6a0 0x7fa3ec0cc790 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009870 tx=0x7fa3dc02b910 comp rx=0 tx=0).stop 2024-03-18T14:42:36.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.540+0000 7fa3f4636640 1 -- 172.21.15.6:0/4101752759 shutdown_connections 2024-03-18T14:42:36.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.540+0000 7fa3f4636640 1 --2- 172.21.15.6:0/4101752759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec0cf6a0 0x7fa3ec0cc790 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.540+0000 7fa3f4636640 1 -- 172.21.15.6:0/4101752759 >> 172.21.15.6:0/4101752759 conn(0x7fa3ec0b35c0 msgr2=0x7fa3ec0b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:36.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.540+0000 7fa3f4636640 1 -- 172.21.15.6:0/4101752759 shutdown_connections 2024-03-18T14:42:36.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.540+0000 7fa3f4636640 1 -- 172.21.15.6:0/4101752759 wait complete. 2024-03-18T14:42:36.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 Processor -- start 2024-03-18T14:42:36.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 -- start start 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec099f20 0x7fa3ec159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec09a300 0x7fa3ec1e4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3ec0cf6a0 0x7fa3ec1e5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ec0cd900 con 0x7fa3ec099f20 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ec0cd780 con 0x7fa3ec09a300 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.541+0000 7fa3f4636640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ec0ae700 con 0x7fa3ec0cf6a0 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec099f20 0x7fa3ec159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec099f20 0x7fa3ec159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36892/0 (socket says 172.21.15.6:36892) 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 -- 172.21.15.6:0/2016221547 learned_addr learned my addr 172.21.15.6:0/2016221547 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f1bd1640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec09a300 0x7fa3ec1e4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f2bd3640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3ec0cf6a0 0x7fa3ec1e5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 -- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3ec0cf6a0 msgr2=0x7fa3ec1e5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3ec0cf6a0 0x7fa3ec1e5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 -- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec09a300 msgr2=0x7fa3ec1e4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec09a300 0x7fa3ec1e4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 -- 172.21.15.6:0/2016221547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec06c400 con 0x7fa3ec099f20 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f1bd1640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec09a300 0x7fa3ec1e4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f23d2640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec099f20 0x7fa3ec159fd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c450 tx=0x7fa3e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3e2ffd640 1 -- 172.21.15.6:0/2016221547 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e80052d0 con 0x7fa3ec099f20 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3e2ffd640 1 -- 172.21.15.6:0/2016221547 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3e8005470 con 0x7fa3ec099f20 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ec06c6e0 con 0x7fa3ec099f20 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3e2ffd640 1 -- 172.21.15.6:0/2016221547 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8012870 con 0x7fa3ec099f20 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.542+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3ec06cca0 con 0x7fa3ec099f20 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.543+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3ec0b7960 con 0x7fa3ec099f20 2024-03-18T14:42:36.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.543+0000 7fa3e2ffd640 1 -- 172.21.15.6:0/2016221547 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3e8012a10 con 0x7fa3ec099f20 2024-03-18T14:42:36.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.544+0000 7fa3e2ffd640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3d006bbd0 0x7fa3d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.544+0000 7fa3e2ffd640 1 -- 172.21.15.6:0/2016221547 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa3e808aa00 con 0x7fa3ec099f20 2024-03-18T14:42:36.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.544+0000 7fa3f1bd1640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3d006bbd0 0x7fa3d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.545+0000 7fa3f1bd1640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3d006bbd0 0x7fa3d006e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc036000 tx=0x7fa3dc03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.548+0000 7fa3e2ffd640 1 -- 172.21.15.6:0/2016221547 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3e8059180 con 0x7fa3ec099f20 2024-03-18T14:42:36.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.732+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa3ec0751b0 con 0x7fa3ec099f20 2024-03-18T14:42:36.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.733+0000 7fa3e2ffd640 1 -- 172.21.15.6:0/2016221547 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fa3e8058a80 con 0x7fa3ec099f20 2024-03-18T14:42:36.735 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:36.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3d006bbd0 msgr2=0x7fa3d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3d006bbd0 0x7fa3d006e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc036000 tx=0x7fa3dc03c040 comp rx=0 tx=0).stop 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec099f20 msgr2=0x7fa3ec159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec099f20 0x7fa3ec159fd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c450 tx=0x7fa3e800c910 comp rx=0 tx=0).stop 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 shutdown_connections 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa3d006bbd0 0x7fa3d006e090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3ec0cf6a0 0x7fa3ec1e5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3ec09a300 0x7fa3ec1e4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 --2- 172.21.15.6:0/2016221547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3ec099f20 0x7fa3ec159fd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 >> 172.21.15.6:0/2016221547 conn(0x7fa3ec0b35c0 msgr2=0x7fa3ec0af780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 shutdown_connections 2024-03-18T14:42:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.735+0000 7fa3f4636640 1 -- 172.21.15.6:0/2016221547 wait complete. 2024-03-18T14:42:36.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.856+0000 7f7071c01640 1 Processor -- start 2024-03-18T14:42:36.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.856+0000 7f7071c01640 1 -- start start 2024-03-18T14:42:36.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f7071c01640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f706c0d3570 0x7f706c0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f7071c01640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d0450 0x7f706c0d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f7071c01640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f706c0d39d0 con 0x7f706c15e030 2024-03-18T14:42:36.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f7071c01640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f706c15e410 con 0x7f706c0d3570 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f7071c01640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f706c0ce300 con 0x7f706c0d0450 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f706affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f706c0d3570 0x7f706c0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f706b7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f706c15e030 0x7f706c0ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48498/0 (socket says 172.21.15.6:48498) 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f706b7fe640 1 -- 172.21.15.6:0/3911814177 learned_addr learned my addr 172.21.15.6:0/3911814177 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.857+0000 7f706bfff640 1 --2- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d0450 0x7f706c0d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.858+0000 7f706affd640 1 -- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d0450 msgr2=0x7f706c0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.858+0000 7f706affd640 1 --2- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d0450 0x7f706c0d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.858+0000 7f706affd640 1 -- 172.21.15.6:0/3911814177 >> v1:172.21.15.92:6789/0 conn(0x7f706c15e030 legacy=0x7f706c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.858+0000 7f706a7fc640 1 -- 172.21.15.6:0/3911814177 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688386969 0 0) 0x7f706c0d39d0 con 0x7f706c15e030 2024-03-18T14:42:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.858+0000 7f706affd640 1 -- 172.21.15.6:0/3911814177 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c0d0e20 con 0x7f706c0d3570 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.858+0000 7f706affd640 1 --2- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c0d3570 0x7f706c0b6140 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f70600097a0 tx=0x7f706002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da345912dcb9e8ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f706a7fc640 1 -- 172.21.15.6:0/3911814177 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060035020 con 0x7f706c0d3570 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f706a7fc640 1 -- 172.21.15.6:0/3911814177 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7060031830 con 0x7f706c0d3570 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f706a7fc640 1 -- 172.21.15.6:0/3911814177 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060031c00 con 0x7f706c0d3570 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 -- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c0d3570 msgr2=0x7f706c0b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 --2- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c0d3570 0x7f706c0b6140 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f70600097a0 tx=0x7f706002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 -- 172.21.15.6:0/3911814177 shutdown_connections 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 --2- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d0450 0x7f706c0d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 --2- 172.21.15.6:0/3911814177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c0d3570 0x7f706c0b6140 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 -- 172.21.15.6:0/3911814177 >> 172.21.15.6:0/3911814177 conn(0x7f706c06c8c0 msgr2=0x7f706c06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:36.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 -- 172.21.15.6:0/3911814177 shutdown_connections 2024-03-18T14:42:36.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.859+0000 7f7071c01640 1 -- 172.21.15.6:0/3911814177 wait complete. 2024-03-18T14:42:36.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.860+0000 7f7071c01640 1 Processor -- start 2024-03-18T14:42:36.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f7071c01640 1 -- start start 2024-03-18T14:42:36.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f7071c01640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f706c0d0450 0x7f706c14c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f7071c01640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d3570 0x7f706c0cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f7071c01640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c15e030 0x7f706c0cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f7071c01640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f706c0ce550 con 0x7f706c0d0450 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f7071c01640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f706c0d3290 con 0x7f706c15e030 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f7071c01640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f706c0ce830 con 0x7f706c0d3570 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f706b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f706c0d0450 0x7f706c14c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f706b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f706c0d0450 0x7f706c14c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36904/0 (socket says 172.21.15.6:36904) 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f706b7fe640 1 -- 172.21.15.6:0/1099339557 learned_addr learned my addr 172.21.15.6:0/1099339557 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f706bfff640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c15e030 0x7f706c0cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.861+0000 7f706affd640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d3570 0x7f706c0cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f706b7fe640 1 -- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d3570 msgr2=0x7f706c0cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f706b7fe640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d3570 0x7f706c0cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f706b7fe640 1 -- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c15e030 msgr2=0x7f706c0cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f706b7fe640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c15e030 0x7f706c0cc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f706b7fe640 1 -- 172.21.15.6:0/1099339557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c15b000 con 0x7f706c0d0450 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f706b7fe640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f706c0d0450 0x7f706c14c970 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f705c00c460 tx=0x7f705c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f7053fff640 1 -- 172.21.15.6:0/1099339557 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c018070 con 0x7f706c0d0450 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f7053fff640 1 -- 172.21.15.6:0/1099339557 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f705c007500 con 0x7f706c0d0450 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706c0cc9b0 con 0x7f706c0d0450 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f7053fff640 1 -- 172.21.15.6:0/1099339557 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c004a20 con 0x7f706c0d0450 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.862+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c0cd250 con 0x7f706c0d0450 2024-03-18T14:42:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.863+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706c0ccda0 con 0x7f706c0d0450 2024-03-18T14:42:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.864+0000 7f7053fff640 1 -- 172.21.15.6:0/1099339557 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f705c004bc0 con 0x7f706c0d0450 2024-03-18T14:42:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.864+0000 7f7053fff640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f705806bbd0 0x7f705806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.864+0000 7f7053fff640 1 -- 172.21.15.6:0/1099339557 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f705c08a160 con 0x7f706c0d0450 2024-03-18T14:42:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.864+0000 7f706affd640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f705806bbd0 0x7f705806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.865+0000 7f706affd640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f705806bbd0 0x7f705806e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7060036000 tx=0x7f706003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:36.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:36.868+0000 7f7053fff640 1 -- 172.21.15.6:0/1099339557 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f706c15b000 con 0x7f706c0d0450 2024-03-18T14:42:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.049+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f706c0751b0 con 0x7f706c0d0450 2024-03-18T14:42:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.049+0000 7f7053fff640 1 -- 172.21.15.6:0/1099339557 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f705c058180 con 0x7f706c0d0450 2024-03-18T14:42:37.052 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f705806bbd0 msgr2=0x7f705806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f705806bbd0 0x7f705806e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7060036000 tx=0x7f706003d040 comp rx=0 tx=0).stop 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f706c0d0450 msgr2=0x7f706c14c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f706c0d0450 0x7f706c14c970 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f705c00c460 tx=0x7f705c00c920 comp rx=0 tx=0).stop 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 shutdown_connections 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f705806bbd0 0x7f705806e090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f706c15e030 0x7f706c0cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706c0d3570 0x7f706c0cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 --2- 172.21.15.6:0/1099339557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f706c0d0450 0x7f706c14c970 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 >> 172.21.15.6:0/1099339557 conn(0x7f706c06c8c0 msgr2=0x7f706c06cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 shutdown_connections 2024-03-18T14:42:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.051+0000 7f7071c01640 1 -- 172.21.15.6:0/1099339557 wait complete. 2024-03-18T14:42:37.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.166+0000 7fbb3913b640 1 Processor -- start 2024-03-18T14:42:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.166+0000 7fbb3913b640 1 -- start start 2024-03-18T14:42:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.166+0000 7fbb3913b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbb3415dfd0 0x7fbb340b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.166+0000 7fbb3913b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb340b5740 0x7fbb340b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.166+0000 7fbb3913b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3415e3b0 con 0x7fbb340d1e40 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.166+0000 7fbb3913b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbb340b60e0 con 0x7fbb3415dfd0 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.166+0000 7fbb3913b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbb340b8470 con 0x7fbb340b5740 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb3359e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbb340d1e40 0x7fbb340d22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50002/0 (socket says 172.21.15.6:50002) 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb3359e640 1 -- 172.21.15.6:0/518638299 learned_addr learned my addr 172.21.15.6:0/518638299 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb2bfff640 1 --2- 172.21.15.6:0/518638299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb340b5740 0x7fbb340b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb32d9d640 1 --2- 172.21.15.6:0/518638299 >> v2:172.21.15.92:3300/0 conn(0x7fbb3415dfd0 0x7fbb340b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb3259c640 1 -- 172.21.15.6:0/518638299 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 582893013 0 0) 0x7fbb3415e3b0 con 0x7fbb340d1e40 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb3259c640 1 -- 172.21.15.6:0/518638299 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb20003520 con 0x7fbb340d1e40 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb2bfff640 1 -- 172.21.15.6:0/518638299 >> v2:172.21.15.92:3300/0 conn(0x7fbb3415dfd0 msgr2=0x7fbb340b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb2bfff640 1 --2- 172.21.15.6:0/518638299 >> v2:172.21.15.92:3300/0 conn(0x7fbb3415dfd0 0x7fbb340b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb2bfff640 1 -- 172.21.15.6:0/518638299 >> v1:172.21.15.6:6789/0 conn(0x7fbb340d1e40 legacy=0x7fbb340d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.167+0000 7fbb2bfff640 1 -- 172.21.15.6:0/518638299 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb340b8750 con 0x7fbb340b5740 2024-03-18T14:42:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.168+0000 7fbb2bfff640 1 --2- 172.21.15.6:0/518638299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb340b5740 0x7fbb340b5ba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0097a0 tx=0x7fbb1c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbd21c7c8f536375 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.168+0000 7fbb3259c640 1 -- 172.21.15.6:0/518638299 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c035020 con 0x7fbb340b5740 2024-03-18T14:42:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.168+0000 7fbb3259c640 1 -- 172.21.15.6:0/518638299 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb1c031830 con 0x7fbb340b5740 2024-03-18T14:42:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.168+0000 7fbb3259c640 1 -- 172.21.15.6:0/518638299 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c031c00 con 0x7fbb340b5740 2024-03-18T14:42:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.168+0000 7fbb3913b640 1 -- 172.21.15.6:0/518638299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb340b5740 msgr2=0x7fbb340b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.168+0000 7fbb3913b640 1 --2- 172.21.15.6:0/518638299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb340b5740 0x7fbb340b5ba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0097a0 tx=0x7fbb1c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.169+0000 7fbb3913b640 1 -- 172.21.15.6:0/518638299 shutdown_connections 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.169+0000 7fbb3913b640 1 --2- 172.21.15.6:0/518638299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb340b5740 0x7fbb340b5ba0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.169+0000 7fbb3913b640 1 --2- 172.21.15.6:0/518638299 >> v2:172.21.15.92:3300/0 conn(0x7fbb3415dfd0 0x7fbb340b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.169+0000 7fbb3913b640 1 -- 172.21.15.6:0/518638299 >> 172.21.15.6:0/518638299 conn(0x7fbb34072d10 msgr2=0x7fbb34073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.169+0000 7fbb3913b640 1 -- 172.21.15.6:0/518638299 shutdown_connections 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.169+0000 7fbb3913b640 1 -- 172.21.15.6:0/518638299 wait complete. 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.170+0000 7fbb3913b640 1 Processor -- start 2024-03-18T14:42:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.170+0000 7fbb3913b640 1 -- start start 2024-03-18T14:42:37.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.170+0000 7fbb3913b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb340b5740 0x7fbb3415f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.170+0000 7fbb3913b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb340d1e40 0x7fbb340ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb3913b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3415dfd0 0x7fbb3414fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb3913b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbb340b9b90 con 0x7fbb340b5740 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb3913b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbb340b8d10 con 0x7fbb340d1e40 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb3913b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbb340b7e00 con 0x7fbb3415dfd0 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb340b5740 0x7fbb3415f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb340b5740 0x7fbb3415f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36916/0 (socket says 172.21.15.6:36916) 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 -- 172.21.15.6:0/700609161 learned_addr learned my addr 172.21.15.6:0/700609161 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb2bfff640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb340d1e40 0x7fbb340ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb3359e640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3415dfd0 0x7fbb3414fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 -- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3415dfd0 msgr2=0x7fbb3414fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3415dfd0 0x7fbb3414fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 -- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb340d1e40 msgr2=0x7fbb340ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb340d1e40 0x7fbb340ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 -- 172.21.15.6:0/700609161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3415f9d0 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb32d9d640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb340b5740 0x7fbb3415f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400e8a0 tx=0x7fbb2400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb2b7fe640 1 -- 172.21.15.6:0/700609161 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24017070 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb2b7fe640 1 -- 172.21.15.6:0/700609161 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2400c940 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3415fcb0 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb2b7fe640 1 -- 172.21.15.6:0/700609161 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2400cd30 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.171+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb34160550 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.172+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3415fe80 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.173+0000 7fbb2b7fe640 1 -- 172.21.15.6:0/700609161 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbb2401a070 con 0x7fbb340b5740 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.173+0000 7fbb2b7fe640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbb1406bbd0 0x7fbb1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.173+0000 7fbb2b7fe640 1 -- 172.21.15.6:0/700609161 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbb2408afd0 con 0x7fbb340b5740 2024-03-18T14:42:37.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.173+0000 7fbb2bfff640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbb1406bbd0 0x7fbb1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.174+0000 7fbb2bfff640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbb1406bbd0 0x7fbb1406e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c036000 tx=0x7fbb1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.177+0000 7fbb2b7fe640 1 -- 172.21.15.6:0/700609161 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbb24059100 con 0x7fbb340b5740 2024-03-18T14:42:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.359+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbb340d04c0 con 0x7fbb340b5740 2024-03-18T14:42:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.359+0000 7fbb2b7fe640 1 -- 172.21.15.6:0/700609161 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fbb340d04c0 con 0x7fbb340b5740 2024-03-18T14:42:37.362 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbb1406bbd0 msgr2=0x7fbb1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbb1406bbd0 0x7fbb1406e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c036000 tx=0x7fbb1c008030 comp rx=0 tx=0).stop 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb340b5740 msgr2=0x7fbb3415f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb340b5740 0x7fbb3415f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400e8a0 tx=0x7fbb2400ed60 comp rx=0 tx=0).stop 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 shutdown_connections 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbb1406bbd0 0x7fbb1406e090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3415dfd0 0x7fbb3414fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb340d1e40 0x7fbb340ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 --2- 172.21.15.6:0/700609161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb340b5740 0x7fbb3415f0d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 >> 172.21.15.6:0/700609161 conn(0x7fbb34072d10 msgr2=0x7fbb340b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 shutdown_connections 2024-03-18T14:42:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.361+0000 7fbb3913b640 1 -- 172.21.15.6:0/700609161 wait complete. 2024-03-18T14:42:37.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:37.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.476+0000 7fcc2e263640 1 Processor -- start 2024-03-18T14:42:37.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.477+0000 7fcc2e263640 1 -- start start 2024-03-18T14:42:37.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.477+0000 7fcc2e263640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc2815e030 0x7fcc280d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.477+0000 7fcc2e263640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcc280b6140 0x7fcc280ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.477+0000 7fcc2e263640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc280b65a0 con 0x7fcc280d0450 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.477+0000 7fcc2e263640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcc2815e410 con 0x7fcc280b6140 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.477+0000 7fcc2e263640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcc280ce300 con 0x7fcc2815e030 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc27fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc2815e030 0x7fcc280d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc2c800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fcc280d0450 0x7fcc280d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48506/0 (socket says 172.21.15.6:48506) 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc277fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcc280b6140 0x7fcc280ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc2c800640 1 -- 172.21.15.6:0/696439075 learned_addr learned my addr 172.21.15.6:0/696439075 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc26ffd640 1 -- 172.21.15.6:0/696439075 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854947559 0 0) 0x7fcc280b65a0 con 0x7fcc280d0450 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc27fff640 1 -- 172.21.15.6:0/696439075 >> v2:172.21.15.92:3300/0 conn(0x7fcc280b6140 msgr2=0x7fcc280ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc27fff640 1 --2- 172.21.15.6:0/696439075 >> v2:172.21.15.92:3300/0 conn(0x7fcc280b6140 0x7fcc280ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc27fff640 1 -- 172.21.15.6:0/696439075 >> v1:172.21.15.92:6789/0 conn(0x7fcc280d0450 legacy=0x7fcc280d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.478+0000 7fcc27fff640 1 -- 172.21.15.6:0/696439075 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc280d2fd0 con 0x7fcc2815e030 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc27fff640 1 --2- 172.21.15.6:0/696439075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc2815e030 0x7fcc280d3570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc180097a0 tx=0x7fcc1802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab923d9b6dcf1ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc26ffd640 1 -- 172.21.15.6:0/696439075 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18035020 con 0x7fcc2815e030 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc26ffd640 1 -- 172.21.15.6:0/696439075 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc18031830 con 0x7fcc2815e030 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc26ffd640 1 -- 172.21.15.6:0/696439075 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18031c00 con 0x7fcc2815e030 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 -- 172.21.15.6:0/696439075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc2815e030 msgr2=0x7fcc280d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 --2- 172.21.15.6:0/696439075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc2815e030 0x7fcc280d3570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc180097a0 tx=0x7fcc1802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 -- 172.21.15.6:0/696439075 shutdown_connections 2024-03-18T14:42:37.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 --2- 172.21.15.6:0/696439075 >> v2:172.21.15.92:3300/0 conn(0x7fcc280b6140 0x7fcc280ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 --2- 172.21.15.6:0/696439075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc2815e030 0x7fcc280d3570 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 -- 172.21.15.6:0/696439075 >> 172.21.15.6:0/696439075 conn(0x7fcc2806c8c0 msgr2=0x7fcc2806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:37.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 -- 172.21.15.6:0/696439075 shutdown_connections 2024-03-18T14:42:37.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.479+0000 7fcc2e263640 1 -- 172.21.15.6:0/696439075 wait complete. 2024-03-18T14:42:37.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.480+0000 7fcc2e263640 1 Processor -- start 2024-03-18T14:42:37.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc2e263640 1 -- start start 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc2e263640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc280b6140 0x7fcc280b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc2e263640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc280d0450 0x7fcc280cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc2e263640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2815e030 0x7fcc2814ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc2e263640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcc280ce830 con 0x7fcc280b6140 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc2e263640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc280ce550 con 0x7fcc2815e030 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc2e263640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcc280cf400 con 0x7fcc280d0450 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.481+0000 7fcc27fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc280b6140 0x7fcc280b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc280b6140 0x7fcc280b5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36924/0 (socket says 172.21.15.6:36924) 2024-03-18T14:42:37.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 -- 172.21.15.6:0/2310931644 learned_addr learned my addr 172.21.15.6:0/2310931644 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc277fe640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc280d0450 0x7fcc280cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc2c800640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2815e030 0x7fcc2814ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 -- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc280d0450 msgr2=0x7fcc280cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc280d0450 0x7fcc280cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 -- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2815e030 msgr2=0x7fcc2814ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2815e030 0x7fcc2814ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 -- 172.21.15.6:0/2310931644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2815f380 con 0x7fcc280b6140 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc27fff640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc280b6140 0x7fcc280b5820 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc180098c0 tx=0x7fcc180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc24ff9640 1 -- 172.21.15.6:0/2310931644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18041070 con 0x7fcc280b6140 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc24ff9640 1 -- 172.21.15.6:0/2310931644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc1803b920 con 0x7fcc280b6140 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc24ff9640 1 -- 172.21.15.6:0/2310931644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18035040 con 0x7fcc280b6140 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2815f660 con 0x7fcc280b6140 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.482+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2815fc20 con 0x7fcc280b6140 2024-03-18T14:42:37.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.483+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc280751b0 con 0x7fcc280b6140 2024-03-18T14:42:37.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.484+0000 7fcc24ff9640 1 -- 172.21.15.6:0/2310931644 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcc1804c060 con 0x7fcc280b6140 2024-03-18T14:42:37.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.484+0000 7fcc24ff9640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc1406bbd0 0x7fcc1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.484+0000 7fcc24ff9640 1 -- 172.21.15.6:0/2310931644 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcc180ac630 con 0x7fcc280b6140 2024-03-18T14:42:37.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.484+0000 7fcc277fe640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc1406bbd0 0x7fcc1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.485+0000 7fcc277fe640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc1406bbd0 0x7fcc1406e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc280ca830 tx=0x7fcc1c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.488+0000 7fcc24ff9640 1 -- 172.21.15.6:0/2310931644 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcc18031800 con 0x7fcc280b6140 2024-03-18T14:42:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.672+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcc2815b480 con 0x7fcc280b6140 2024-03-18T14:42:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.672+0000 7fcc24ff9640 1 -- 172.21.15.6:0/2310931644 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fcc1807a6d0 con 0x7fcc280b6140 2024-03-18T14:42:37.675 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc1406bbd0 msgr2=0x7fcc1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc1406bbd0 0x7fcc1406e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc280ca830 tx=0x7fcc1c009190 comp rx=0 tx=0).stop 2024-03-18T14:42:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc280b6140 msgr2=0x7fcc280b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc280b6140 0x7fcc280b5820 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc180098c0 tx=0x7fcc180046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 shutdown_connections 2024-03-18T14:42:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcc1406bbd0 0x7fcc1406e090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc2815e030 0x7fcc2814ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc280d0450 0x7fcc280cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 --2- 172.21.15.6:0/2310931644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc280b6140 0x7fcc280b5820 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 >> 172.21.15.6:0/2310931644 conn(0x7fcc2806c8c0 msgr2=0x7fcc280b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 shutdown_connections 2024-03-18T14:42:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.674+0000 7fcc2e263640 1 -- 172.21.15.6:0/2310931644 wait complete. 2024-03-18T14:42:37.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:37.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.788+0000 7f4622436640 1 Processor -- start 2024-03-18T14:42:37.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f4622436640 1 -- start start 2024-03-18T14:42:37.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f4622436640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f461c15e120 0x7f461c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f4622436640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461c0b20a0 con 0x7f461c162700 2024-03-18T14:42:37.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f4622436640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461c161c20 con 0x7f461c06b8d0 2024-03-18T14:42:37.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f4622436640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f461c0b2380 con 0x7f461c15e120 2024-03-18T14:42:37.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f461bfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f461c15e120 0x7f461c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f461b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f461c162700 0x7f461c06b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50018/0 (socket says 172.21.15.6:50018) 2024-03-18T14:42:37.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f461b7fe640 1 -- 172.21.15.6:0/2125128112 learned_addr learned my addr 172.21.15.6:0/2125128112 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:37.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.789+0000 7f461bfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f461c15e120 0x7f461c0cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36942/0 (socket says 172.21.15.6:36942) 2024-03-18T14:42:37.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.790+0000 7f461bfff640 1 -- 172.21.15.6:0/2125128112 >> v1:172.21.15.92:6789/0 conn(0x7f461c06b8d0 legacy=0x7f461c0cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.790+0000 7f461bfff640 1 -- 172.21.15.6:0/2125128112 >> v1:172.21.15.6:6789/0 conn(0x7f461c162700 legacy=0x7f461c06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.790+0000 7f461bfff640 1 -- 172.21.15.6:0/2125128112 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c161f00 con 0x7f461c15e120 2024-03-18T14:42:37.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.790+0000 7f461affd640 1 -- 172.21.15.6:0/2125128112 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883744417 0 0) 0x7f461c0b20a0 con 0x7f461c162700 2024-03-18T14:42:37.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.790+0000 7f461bfff640 1 --2- 172.21.15.6:0/2125128112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c15e120 0x7f461c0cde20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f46100097a0 tx=0x7f461002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31b37c4336a2f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f461affd640 1 -- 172.21.15.6:0/2125128112 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461002f810 con 0x7f461c15e120 2024-03-18T14:42:37.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f461affd640 1 -- 172.21.15.6:0/2125128112 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f461002f9b0 con 0x7f461c15e120 2024-03-18T14:42:37.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f461affd640 1 -- 172.21.15.6:0/2125128112 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461002fd80 con 0x7f461c15e120 2024-03-18T14:42:37.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f4622436640 1 -- 172.21.15.6:0/2125128112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c15e120 msgr2=0x7f461c0cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f4622436640 1 --2- 172.21.15.6:0/2125128112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c15e120 0x7f461c0cde20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f46100097a0 tx=0x7f461002b9f0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f4622436640 1 -- 172.21.15.6:0/2125128112 shutdown_connections 2024-03-18T14:42:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f4622436640 1 --2- 172.21.15.6:0/2125128112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c15e120 0x7f461c0cde20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f4622436640 1 -- 172.21.15.6:0/2125128112 >> 172.21.15.6:0/2125128112 conn(0x7f461c099a50 msgr2=0x7f461c099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f4622436640 1 -- 172.21.15.6:0/2125128112 shutdown_connections 2024-03-18T14:42:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.791+0000 7f4622436640 1 -- 172.21.15.6:0/2125128112 wait complete. 2024-03-18T14:42:37.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.792+0000 7f4622436640 1 Processor -- start 2024-03-18T14:42:37.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f4622436640 1 -- start start 2024-03-18T14:42:37.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f4622436640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c06b8d0 0x7f461c0c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f4622436640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461c162700 0x7f461c0c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f4622436640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461c0b5ef0 0x7f461c0c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f4622436640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f461c0b1af0 con 0x7f461c06b8d0 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f4622436640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f461c0ce360 con 0x7f461c0b5ef0 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f4622436640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f461c0b1c70 con 0x7f461c162700 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f461bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c06b8d0 0x7f461c0c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f461bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c06b8d0 0x7f461c0c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36952/0 (socket says 172.21.15.6:36952) 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f461bfff640 1 -- 172.21.15.6:0/515075514 learned_addr learned my addr 172.21.15.6:0/515075514 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f46209d3640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461c0b5ef0 0x7f461c0c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.793+0000 7f461b7fe640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461c162700 0x7f461c0c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f461bfff640 1 -- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461c162700 msgr2=0x7f461c0c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f461bfff640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461c162700 0x7f461c0c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f461bfff640 1 -- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461c0b5ef0 msgr2=0x7f461c0c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f461bfff640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461c0b5ef0 0x7f461c0c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f461bfff640 1 -- 172.21.15.6:0/515075514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c0c4870 con 0x7f461c06b8d0 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f461bfff640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c06b8d0 0x7f461c0c5470 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f46100098c0 tx=0x7f4610004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f4618ff9640 1 -- 172.21.15.6:0/515075514 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610047070 con 0x7f461c06b8d0 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f4618ff9640 1 -- 172.21.15.6:0/515075514 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4610037c60 con 0x7f461c06b8d0 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c0c8180 con 0x7f461c06b8d0 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f4618ff9640 1 -- 172.21.15.6:0/515075514 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461002fe60 con 0x7f461c06b8d0 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.794+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c0c8740 con 0x7f461c06b8d0 2024-03-18T14:42:37.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.795+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461c0cebc0 con 0x7f461c06b8d0 2024-03-18T14:42:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.795+0000 7f4618ff9640 1 -- 172.21.15.6:0/515075514 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f461004a070 con 0x7f461c06b8d0 2024-03-18T14:42:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.796+0000 7f4618ff9640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f460806bbd0 0x7f460806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.796+0000 7f4618ff9640 1 -- 172.21.15.6:0/515075514 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f46100acb00 con 0x7f461c06b8d0 2024-03-18T14:42:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.796+0000 7f461b7fe640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f460806bbd0 0x7f460806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:37.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.796+0000 7f461b7fe640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f460806bbd0 0x7f460806e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f460c007f70 tx=0x7f460c007f00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:37.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.800+0000 7f4618ff9640 1 -- 172.21.15.6:0/515075514 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f461c0cebc0 con 0x7f461c06b8d0 2024-03-18T14:42:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.995+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f461c06bfd0 con 0x7f461c06b8d0 2024-03-18T14:42:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.995+0000 7f4618ff9640 1 -- 172.21.15.6:0/515075514 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f4610033030 con 0x7f461c06b8d0 2024-03-18T14:42:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f460806bbd0 msgr2=0x7f460806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f460806bbd0 0x7f460806e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f460c007f70 tx=0x7f460c007f00 comp rx=0 tx=0).stop 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c06b8d0 msgr2=0x7f461c0c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c06b8d0 0x7f461c0c5470 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f46100098c0 tx=0x7f4610004730 comp rx=0 tx=0).stop 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 shutdown_connections 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f460806bbd0 0x7f460806e090 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461c0b5ef0 0x7f461c0c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461c162700 0x7f461c0c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 --2- 172.21.15.6:0/515075514 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f461c06b8d0 0x7f461c0c5470 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 >> 172.21.15.6:0/515075514 conn(0x7f461c099a50 msgr2=0x7f461c0ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 shutdown_connections 2024-03-18T14:42:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:37.997+0000 7f4622436640 1 -- 172.21.15.6:0/515075514 wait complete. 2024-03-18T14:42:38.011 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:42:38.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:38.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-18T14:42:38.037 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:38.037 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:42:38.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:38.038 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T14:42:38.092 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:42:38.092 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:38.107 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:42:38.107 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:38.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:38.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.261+0000 7fc2bace9640 1 Processor -- start 2024-03-18T14:42:38.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.261+0000 7fc2bace9640 1 -- start start 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2bace9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2b40d17f0 0x7fc2b40b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2bace9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40d05d0 0x7fc2b40d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2bace9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2b40b1eb0 con 0x7fc2b40ce530 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2bace9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b40b2190 con 0x7fc2b40d17f0 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2bace9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b40b2470 con 0x7fc2b40d05d0 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40d05d0 0x7fc2b40d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b8a85640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2b40d17f0 0x7fc2b40b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40d05d0 0x7fc2b40d1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36362/0 (socket says 172.21.15.6:36362) 2024-03-18T14:42:38.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b9286640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc2b40ce530 0x7fc2b40ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48524/0 (socket says 172.21.15.6:48524) 2024-03-18T14:42:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b3fff640 1 -- 172.21.15.6:0/2485983773 learned_addr learned my addr 172.21.15.6:0/2485983773 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b8a85640 1 -- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40d05d0 msgr2=0x7fc2b40d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b8a85640 1 --2- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40d05d0 0x7fc2b40d1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b8a85640 1 -- 172.21.15.6:0/2485983773 >> v1:172.21.15.92:6789/0 conn(0x7fc2b40ce530 legacy=0x7fc2b40ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b8a85640 1 -- 172.21.15.6:0/2485983773 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b40b2750 con 0x7fc2b40d17f0 2024-03-18T14:42:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b37fe640 1 -- 172.21.15.6:0/2485983773 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1865054114 0 0) 0x7fc2b40b1eb0 con 0x7fc2b40ce530 2024-03-18T14:42:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.262+0000 7fc2b3fff640 1 --2- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40d05d0 0x7fc2b40d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:38.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.263+0000 7fc2b8a85640 1 --2- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40b17b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80097a0 tx=0x7fc2a802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caac29faaf42b24a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.263+0000 7fc2b37fe640 1 -- 172.21.15.6:0/2485983773 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8035020 con 0x7fc2b40d17f0 2024-03-18T14:42:38.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.263+0000 7fc2b37fe640 1 -- 172.21.15.6:0/2485983773 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a8031830 con 0x7fc2b40d17f0 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.263+0000 7fc2b37fe640 1 -- 172.21.15.6:0/2485983773 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8031c00 con 0x7fc2b40d17f0 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 -- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d17f0 msgr2=0x7fc2b40b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40b17b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80097a0 tx=0x7fc2a802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 -- 172.21.15.6:0/2485983773 shutdown_connections 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40d05d0 0x7fc2b40d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2485983773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40b17b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 -- 172.21.15.6:0/2485983773 >> 172.21.15.6:0/2485983773 conn(0x7fc2b408b130 msgr2=0x7fc2b408b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 -- 172.21.15.6:0/2485983773 shutdown_connections 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.264+0000 7fc2bace9640 1 -- 172.21.15.6:0/2485983773 wait complete. 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.265+0000 7fc2bace9640 1 Processor -- start 2024-03-18T14:42:38.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.265+0000 7fc2bace9640 1 -- start start 2024-03-18T14:42:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2bace9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40ce530 0x7fc2b41f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2bace9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d05d0 0x7fc2b415c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2bace9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2bace9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b40d2110 con 0x7fc2b40d17f0 2024-03-18T14:42:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2bace9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b415c080 con 0x7fc2b40d05d0 2024-03-18T14:42:38.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2bace9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b40d0a30 con 0x7fc2b40ce530 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b9286640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b8a85640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40ce530 0x7fc2b41f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d05d0 0x7fc2b415c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b9286640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36968/0 (socket says 172.21.15.6:36968) 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b9286640 1 -- 172.21.15.6:0/2539693 learned_addr learned my addr 172.21.15.6:0/2539693 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b8a85640 1 -- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d05d0 msgr2=0x7fc2b415c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b8a85640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d05d0 0x7fc2b415c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b8a85640 1 -- 172.21.15.6:0/2539693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b40d17f0 msgr2=0x7fc2b40c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b8a85640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b8a85640 1 -- 172.21.15.6:0/2539693 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b40c9da0 con 0x7fc2b40ce530 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b9286640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.266+0000 7fc2b3fff640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d05d0 0x7fc2b415c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.267+0000 7fc2b8a85640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40ce530 0x7fc2b41f6d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8033000 tx=0x7fc2a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.267+0000 7fc2b17fa640 1 -- 172.21.15.6:0/2539693 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8035070 con 0x7fc2b40ce530 2024-03-18T14:42:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.267+0000 7fc2b17fa640 1 -- 172.21.15.6:0/2539693 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a8004230 con 0x7fc2b40ce530 2024-03-18T14:42:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.267+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b40ca080 con 0x7fc2b40ce530 2024-03-18T14:42:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.267+0000 7fc2b17fa640 1 -- 172.21.15.6:0/2539693 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a803a6f0 con 0x7fc2b40ce530 2024-03-18T14:42:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.267+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b40ca640 con 0x7fc2b40ce530 2024-03-18T14:42:38.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.268+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b409fa80 con 0x7fc2b40ce530 2024-03-18T14:42:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.268+0000 7fc2b17fa640 1 -- 172.21.15.6:0/2539693 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc2a8038020 con 0x7fc2b40ce530 2024-03-18T14:42:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.268+0000 7fc2b17fa640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc29806bc00 0x7fc29806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.269+0000 7fc2b17fa640 1 -- 172.21.15.6:0/2539693 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc2a80aca40 con 0x7fc2b40ce530 2024-03-18T14:42:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.269+0000 7fc2b3fff640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc29806bc00 0x7fc29806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.269+0000 7fc2b3fff640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc29806bc00 0x7fc29806e0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009630 tx=0x7fc2a0009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.273+0000 7fc2b17fa640 1 -- 172.21.15.6:0/2539693 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc2a8040070 con 0x7fc2b40ce530 2024-03-18T14:42:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.456+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2b404f620 con 0x7fc2b40ce530 2024-03-18T14:42:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.458+0000 7fc2b17fa640 1 -- 172.21.15.6:0/2539693 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fc2a807aae0 con 0x7fc2b40ce530 2024-03-18T14:42:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:38.460 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc29806bc00 msgr2=0x7fc29806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc29806bc00 0x7fc29806e0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009630 tx=0x7fc2a0009240 comp rx=0 tx=0).stop 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40ce530 msgr2=0x7fc2b41f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40ce530 0x7fc2b41f6d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8033000 tx=0x7fc2a80046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 shutdown_connections 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc29806bc00 0x7fc29806e0c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b40d17f0 0x7fc2b40c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b40d05d0 0x7fc2b415c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 --2- 172.21.15.6:0/2539693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40ce530 0x7fc2b41f6d90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 >> 172.21.15.6:0/2539693 conn(0x7fc2b408b130 msgr2=0x7fc2b40b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:38.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 shutdown_connections 2024-03-18T14:42:38.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.459+0000 7fc2bace9640 1 -- 172.21.15.6:0/2539693 wait complete. 2024-03-18T14:42:38.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:38.473 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:38.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.576+0000 7f4f6f814640 1 Processor -- start 2024-03-18T14:42:38.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.577+0000 7f4f6f814640 1 -- start start 2024-03-18T14:42:38.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.577+0000 7f4f6f814640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f680b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.577+0000 7f4f6f814640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f680d39d0 con 0x7f4f680d0450 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.577+0000 7f4f6f814640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f680d2fd0 con 0x7f4f6815e030 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.577+0000 7f4f6f814640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4f6815e410 con 0x7f4f680d3570 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f6ddb1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f680d0450 0x7f4f680d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50020/0 (socket says 172.21.15.6:50020) 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f6ddb1640 1 -- 172.21.15.6:0/2258682066 learned_addr learned my addr 172.21.15.6:0/2258682066 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f6cdaf640 1 --2- 172.21.15.6:0/2258682066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f680b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f5ffff640 1 -- 172.21.15.6:0/2258682066 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837523864 0 0) 0x7f4f680d39d0 con 0x7f4f680d0450 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f5ffff640 1 -- 172.21.15.6:0/2258682066 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f54003520 con 0x7f4f680d0450 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f6cdaf640 1 -- 172.21.15.6:0/2258682066 >> v1:172.21.15.92:6789/0 conn(0x7f4f6815e030 legacy=0x7f4f680ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f6cdaf640 1 -- 172.21.15.6:0/2258682066 >> v1:172.21.15.6:6789/0 conn(0x7f4f680d0450 legacy=0x7f4f680d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f5ffff640 1 -- 172.21.15.6:0/2258682066 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607847207 0 0) 0x7f4f680d2fd0 con 0x7f4f6815e030 2024-03-18T14:42:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.578+0000 7f4f6cdaf640 1 -- 172.21.15.6:0/2258682066 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f680d3290 con 0x7f4f680d3570 2024-03-18T14:42:38.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f6cdaf640 1 --2- 172.21.15.6:0/2258682066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f680b6140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f580097d0 tx=0x7f4f5802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40e987cd44007170 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f5ffff640 1 -- 172.21.15.6:0/2258682066 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58035020 con 0x7f4f680d3570 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f5ffff640 1 -- 172.21.15.6:0/2258682066 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f58031890 con 0x7f4f680d3570 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f5ffff640 1 -- 172.21.15.6:0/2258682066 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58031c60 con 0x7f4f680d3570 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f6f814640 1 -- 172.21.15.6:0/2258682066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 msgr2=0x7f4f680b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f6f814640 1 --2- 172.21.15.6:0/2258682066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f680b6140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f580097d0 tx=0x7f4f5802b930 comp rx=0 tx=0).stop 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f6f814640 1 -- 172.21.15.6:0/2258682066 shutdown_connections 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f6f814640 1 --2- 172.21.15.6:0/2258682066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f680b6140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.579+0000 7f4f6f814640 1 -- 172.21.15.6:0/2258682066 >> 172.21.15.6:0/2258682066 conn(0x7f4f6806c8c0 msgr2=0x7f4f6806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.580+0000 7f4f6f814640 1 -- 172.21.15.6:0/2258682066 shutdown_connections 2024-03-18T14:42:38.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.580+0000 7f4f6f814640 1 -- 172.21.15.6:0/2258682066 wait complete. 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 Processor -- start 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 -- start start 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f680d0450 0x7f4f68155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f6815c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f6815e030 0x7f4f6815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f680ce830 con 0x7f4f6815e030 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4f680ce300 con 0x7f4f680d0450 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6f814640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4f68161a90 con 0x7f4f680d3570 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.581+0000 7f4f6ddb1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f6815e030 0x7f4f6815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f6815e030 0x7f4f6815cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36970/0 (socket says 172.21.15.6:36970) 2024-03-18T14:42:38.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6cdaf640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f6815c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 -- 172.21.15.6:0/3792155928 learned_addr learned my addr 172.21.15.6:0/3792155928 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6d5b0640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f680d0450 0x7f4f68155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 -- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 msgr2=0x7f4f6815c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f6815c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 -- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f680d0450 msgr2=0x7f4f68155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f680d0450 0x7f4f68155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 -- 172.21.15.6:0/3792155928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f680cccc0 con 0x7f4f6815e030 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6ddb1640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f6815e030 0x7f4f6815cb30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000cb10 tx=0x7f4f6000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f5dffb640 1 -- 172.21.15.6:0/3792155928 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60017070 con 0x7f4f6815e030 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f5dffb640 1 -- 172.21.15.6:0/3792155928 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f6000ac90 con 0x7f4f6815e030 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f680ccfa0 con 0x7f4f6815e030 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f5dffb640 1 -- 172.21.15.6:0/3792155928 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60012640 con 0x7f4f6815e030 2024-03-18T14:42:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.582+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f680cd560 con 0x7f4f6815e030 2024-03-18T14:42:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.583+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f680751b0 con 0x7f4f6815e030 2024-03-18T14:42:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.584+0000 7f4f5dffb640 1 -- 172.21.15.6:0/3792155928 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4f6001a070 con 0x7f4f6815e030 2024-03-18T14:42:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.584+0000 7f4f5dffb640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f4c06bc00 0x7f4f4c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.584+0000 7f4f5dffb640 1 -- 172.21.15.6:0/3792155928 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4f6008aa50 con 0x7f4f6815e030 2024-03-18T14:42:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.584+0000 7f4f6d5b0640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f4c06bc00 0x7f4f4c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.585+0000 7f4f6d5b0640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f4c06bc00 0x7f4f4c06e0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4f64000f60 tx=0x7f4f64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.588+0000 7f4f5dffb640 1 -- 172.21.15.6:0/3792155928 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4f60058b80 con 0x7f4f6815e030 2024-03-18T14:42:38.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.771+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f680d19f0 con 0x7f4f6815e030 2024-03-18T14:42:38.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.772+0000 7f4f5dffb640 1 -- 172.21.15.6:0/3792155928 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f4f6005cc20 con 0x7f4f6815e030 2024-03-18T14:42:38.775 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:38.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f4c06bc00 msgr2=0x7f4f4c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f4c06bc00 0x7f4f4c06e0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4f64000f60 tx=0x7f4f64009040 comp rx=0 tx=0).stop 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f6815e030 msgr2=0x7f4f6815cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f6815e030 0x7f4f6815cb30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000cb10 tx=0x7f4f6000a060 comp rx=0 tx=0).stop 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 shutdown_connections 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4f4c06bc00 0x7f4f4c06e0c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f6815e030 0x7f4f6815cb30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f680d3570 0x7f4f6815c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 --2- 172.21.15.6:0/3792155928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f680d0450 0x7f4f68155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 >> 172.21.15.6:0/3792155928 conn(0x7f4f6806c8c0 msgr2=0x7f4f680d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 shutdown_connections 2024-03-18T14:42:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.774+0000 7f4f6f814640 1 -- 172.21.15.6:0/3792155928 wait complete. 2024-03-18T14:42:38.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.889+0000 7fa479ce8640 1 Processor -- start 2024-03-18T14:42:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.889+0000 7fa479ce8640 1 -- start start 2024-03-18T14:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa479ce8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa47409b070 0x7fa47406c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa479ce8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa47409a3c0 0x7fa47415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa479ce8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa47406d820 con 0x7fa47406d440 2024-03-18T14:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa479ce8640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa4740d0d60 con 0x7fa47409b070 2024-03-18T14:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa479ce8640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa4740d0150 con 0x7fa47409a3c0 2024-03-18T14:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa4737fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa47406d440 0x7fa474099cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50030/0 (socket says 172.21.15.6:50030) 2024-03-18T14:42:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa4737fe640 1 -- 172.21.15.6:0/3699874399 learned_addr learned my addr 172.21.15.6:0/3699874399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa472ffd640 1 --2- 172.21.15.6:0/3699874399 >> v2:172.21.15.6:3300/0 conn(0x7fa47409b070 0x7fa47406c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa473fff640 1 --2- 172.21.15.6:0/3699874399 >> v2:172.21.15.92:3300/0 conn(0x7fa47409a3c0 0x7fa47415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa4727fc640 1 -- 172.21.15.6:0/3699874399 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1117015326 0 0) 0x7fa47406d820 con 0x7fa47406d440 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa4727fc640 1 -- 172.21.15.6:0/3699874399 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa45c003520 con 0x7fa47406d440 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa472ffd640 1 -- 172.21.15.6:0/3699874399 >> v2:172.21.15.92:3300/0 conn(0x7fa47409a3c0 msgr2=0x7fa47415d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa472ffd640 1 --2- 172.21.15.6:0/3699874399 >> v2:172.21.15.92:3300/0 conn(0x7fa47409a3c0 0x7fa47415d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa472ffd640 1 -- 172.21.15.6:0/3699874399 >> v1:172.21.15.6:6789/0 conn(0x7fa47406d440 legacy=0x7fa474099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.890+0000 7fa472ffd640 1 -- 172.21.15.6:0/3699874399 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4740d0430 con 0x7fa47409b070 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.891+0000 7fa472ffd640 1 --2- 172.21.15.6:0/3699874399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409b070 0x7fa47406c4e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa4740d1b20 tx=0x7fa46802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838bd5bd14cdd21c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.891+0000 7fa4727fc640 1 -- 172.21.15.6:0/3699874399 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468035020 con 0x7fa47409b070 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.891+0000 7fa4727fc640 1 -- 172.21.15.6:0/3699874399 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa468031830 con 0x7fa47409b070 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.891+0000 7fa4727fc640 1 -- 172.21.15.6:0/3699874399 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468031c00 con 0x7fa47409b070 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 -- 172.21.15.6:0/3699874399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409b070 msgr2=0x7fa47406c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 --2- 172.21.15.6:0/3699874399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409b070 0x7fa47406c4e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa4740d1b20 tx=0x7fa46802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 -- 172.21.15.6:0/3699874399 shutdown_connections 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 --2- 172.21.15.6:0/3699874399 >> v2:172.21.15.92:3300/0 conn(0x7fa47409a3c0 0x7fa47415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 --2- 172.21.15.6:0/3699874399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409b070 0x7fa47406c4e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 -- 172.21.15.6:0/3699874399 >> 172.21.15.6:0/3699874399 conn(0x7fa4740b3660 msgr2=0x7fa4740b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 -- 172.21.15.6:0/3699874399 shutdown_connections 2024-03-18T14:42:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.892+0000 7fa479ce8640 1 -- 172.21.15.6:0/3699874399 wait complete. 2024-03-18T14:42:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.893+0000 7fa479ce8640 1 Processor -- start 2024-03-18T14:42:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.893+0000 7fa479ce8640 1 -- start start 2024-03-18T14:42:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa479ce8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa47406d440 0x7fa4740c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa479ce8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409a3c0 0x7fa474155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa479ce8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4740c5a30 0x7fa47409d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa479ce8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4740cfd70 con 0x7fa47409a3c0 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa479ce8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa4740d09f0 con 0x7fa47406d440 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa479ce8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa4740cfef0 con 0x7fa4740c5a30 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409a3c0 0x7fa474155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409a3c0 0x7fa474155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36990/0 (socket says 172.21.15.6:36990) 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 -- 172.21.15.6:0/1724352308 learned_addr learned my addr 172.21.15.6:0/1724352308 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa4737fe640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa47406d440 0x7fa4740c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa473fff640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4740c5a30 0x7fa47409d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 -- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4740c5a30 msgr2=0x7fa47409d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4740c5a30 0x7fa47409d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 -- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa47406d440 msgr2=0x7fa4740c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa47406d440 0x7fa4740c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 -- 172.21.15.6:0/1724352308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa474155580 con 0x7fa47409a3c0 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.894+0000 7fa472ffd640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409a3c0 0x7fa474155040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa468000c00 tx=0x7fa4680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.895+0000 7fa45bfff640 1 -- 172.21.15.6:0/1724352308 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468040070 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.895+0000 7fa45bfff640 1 -- 172.21.15.6:0/1724352308 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa46803a910 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.895+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4740c63f0 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.895+0000 7fa45bfff640 1 -- 172.21.15.6:0/1724352308 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468035040 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.895+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4740c69b0 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.896+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4740751b0 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.896+0000 7fa45bfff640 1 -- 172.21.15.6:0/1724352308 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa46804b060 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.896+0000 7fa45bfff640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa46006bbd0 0x7fa46006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.896+0000 7fa45bfff640 1 -- 172.21.15.6:0/1724352308 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa4680ac540 con 0x7fa47409a3c0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.896+0000 7fa4737fe640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa46006bbd0 0x7fa46006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.897+0000 7fa4737fe640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa46006bbd0 0x7fa46006e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa464005880 tx=0x7fa464009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:38.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:38.901+0000 7fa45bfff640 1 -- 172.21.15.6:0/1724352308 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa474155580 con 0x7fa47409a3c0 2024-03-18T14:42:39.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.085+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa47409a7e0 con 0x7fa47409a3c0 2024-03-18T14:42:39.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.085+0000 7fa45bfff640 1 -- 172.21.15.6:0/1724352308 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa46807a550 con 0x7fa47409a3c0 2024-03-18T14:42:39.088 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:39.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa46006bbd0 msgr2=0x7fa46006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa46006bbd0 0x7fa46006e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa464005880 tx=0x7fa464009040 comp rx=0 tx=0).stop 2024-03-18T14:42:39.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409a3c0 msgr2=0x7fa474155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409a3c0 0x7fa474155040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa468000c00 tx=0x7fa4680046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 shutdown_connections 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa46006bbd0 0x7fa46006e090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4740c5a30 0x7fa47409d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa47409a3c0 0x7fa474155040 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 --2- 172.21.15.6:0/1724352308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa47406d440 0x7fa4740c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.087+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 >> 172.21.15.6:0/1724352308 conn(0x7fa4740b3660 msgr2=0x7fa4740cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.088+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 shutdown_connections 2024-03-18T14:42:39.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.088+0000 7fa479ce8640 1 -- 172.21.15.6:0/1724352308 wait complete. 2024-03-18T14:42:39.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:39.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.203+0000 7fed64a63640 1 Processor -- start 2024-03-18T14:42:39.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.203+0000 7fed64a63640 1 -- start start 2024-03-18T14:42:39.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed64a63640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed6015dfd0 0x7fed600b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed64a63640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed600b7e00 0x7fed600b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed64a63640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6015e3b0 con 0x7fed600b5740 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed64a63640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fed600b60e0 con 0x7fed600b7e00 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed64a63640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fed600b8470 con 0x7fed6015dfd0 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed5dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fed600b5740 0x7fed600d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50044/0 (socket says 172.21.15.6:50044) 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed5dd9b640 1 -- 172.21.15.6:0/2246232193 learned_addr learned my addr 172.21.15.6:0/2246232193 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed5e59c640 1 --2- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed6015dfd0 0x7fed600b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.204+0000 7fed5ed9d640 1 --2- 172.21.15.6:0/2246232193 >> v2:172.21.15.92:3300/0 conn(0x7fed600b7e00 0x7fed600b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5d59a640 1 -- 172.21.15.6:0/2246232193 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2707775068 0 0) 0x7fed6015e3b0 con 0x7fed600b5740 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5d59a640 1 -- 172.21.15.6:0/2246232193 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed4c003520 con 0x7fed600b5740 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5ed9d640 1 -- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed6015dfd0 msgr2=0x7fed600b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5ed9d640 1 --2- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed6015dfd0 0x7fed600b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5ed9d640 1 -- 172.21.15.6:0/2246232193 >> v1:172.21.15.6:6789/0 conn(0x7fed600b5740 legacy=0x7fed600d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5ed9d640 1 -- 172.21.15.6:0/2246232193 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed600b8750 con 0x7fed600b7e00 2024-03-18T14:42:39.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5e59c640 1 --2- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed6015dfd0 0x7fed600b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.205+0000 7fed5ed9d640 1 --2- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 0x7fed600b5ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fed580097a0 tx=0x7fed5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=663a5ba769c6e606 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed5d59a640 1 -- 172.21.15.6:0/2246232193 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58035020 con 0x7fed600b7e00 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed5d59a640 1 -- 172.21.15.6:0/2246232193 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed58031830 con 0x7fed600b7e00 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed5d59a640 1 -- 172.21.15.6:0/2246232193 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58031be0 con 0x7fed600b7e00 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 -- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 msgr2=0x7fed600b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 --2- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 0x7fed600b5ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fed580097a0 tx=0x7fed5802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 -- 172.21.15.6:0/2246232193 shutdown_connections 2024-03-18T14:42:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 --2- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 0x7fed600b5ba0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 --2- 172.21.15.6:0/2246232193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed6015dfd0 0x7fed600b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 -- 172.21.15.6:0/2246232193 >> 172.21.15.6:0/2246232193 conn(0x7fed60072d10 msgr2=0x7fed60073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 -- 172.21.15.6:0/2246232193 shutdown_connections 2024-03-18T14:42:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.206+0000 7fed64a63640 1 -- 172.21.15.6:0/2246232193 wait complete. 2024-03-18T14:42:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.207+0000 7fed64a63640 1 Processor -- start 2024-03-18T14:42:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed64a63640 1 -- start start 2024-03-18T14:42:39.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed64a63640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed600b5740 0x7fed6015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed64a63640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 0x7fed600c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed64a63640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6015dfd0 0x7fed601520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed64a63640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fed600b9b90 con 0x7fed6015dfd0 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed64a63640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed600b8d10 con 0x7fed600b7e00 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed64a63640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fed600d2550 con 0x7fed600b5740 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed5ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6015dfd0 0x7fed601520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed5ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6015dfd0 0x7fed601520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37004/0 (socket says 172.21.15.6:37004) 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed5ed9d640 1 -- 172.21.15.6:0/1975297473 learned_addr learned my addr 172.21.15.6:0/1975297473 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed5e59c640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed600b5740 0x7fed6015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.208+0000 7fed5dd9b640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 0x7fed600c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed5ed9d640 1 -- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed600b5740 msgr2=0x7fed6015b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed5ed9d640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed600b5740 0x7fed6015b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed5ed9d640 1 -- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 msgr2=0x7fed600c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed5ed9d640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 0x7fed600c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed5ed9d640 1 -- 172.21.15.6:0/1975297473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed600c7ef0 con 0x7fed6015dfd0 2024-03-18T14:42:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed5ed9d640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6015dfd0 0x7fed601520d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fed5802b910 tx=0x7fed580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed56ffd640 1 -- 172.21.15.6:0/1975297473 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58035040 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed56ffd640 1 -- 172.21.15.6:0/1975297473 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5803b840 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed600c81d0 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed56ffd640 1 -- 172.21.15.6:0/1975297473 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5803bc10 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.209+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed600c8790 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.210+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed600751b0 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.210+0000 7fed56ffd640 1 -- 172.21.15.6:0/1975297473 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fed58038020 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.211+0000 7fed56ffd640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fed4006bbd0 0x7fed4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.211+0000 7fed56ffd640 1 -- 172.21.15.6:0/1975297473 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fed580acdc0 con 0x7fed6015dfd0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.211+0000 7fed5e59c640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fed4006bbd0 0x7fed4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.211+0000 7fed5e59c640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fed4006bbd0 0x7fed4006e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed50009bd0 tx=0x7fed5000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.215+0000 7fed56ffd640 1 -- 172.21.15.6:0/1975297473 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fed5807ade0 con 0x7fed6015dfd0 2024-03-18T14:42:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.398+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed600d2270 con 0x7fed6015dfd0 2024-03-18T14:42:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.398+0000 7fed56ffd640 1 -- 172.21.15.6:0/1975297473 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fed5807ee80 con 0x7fed6015dfd0 2024-03-18T14:42:39.401 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fed4006bbd0 msgr2=0x7fed4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fed4006bbd0 0x7fed4006e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed50009bd0 tx=0x7fed5000b040 comp rx=0 tx=0).stop 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6015dfd0 msgr2=0x7fed601520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6015dfd0 0x7fed601520d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fed5802b910 tx=0x7fed580046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 shutdown_connections 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fed4006bbd0 0x7fed4006e090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed6015dfd0 0x7fed601520d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed600b7e00 0x7fed600c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 --2- 172.21.15.6:0/1975297473 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed600b5740 0x7fed6015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 >> 172.21.15.6:0/1975297473 conn(0x7fed60072d10 msgr2=0x7fed600b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 shutdown_connections 2024-03-18T14:42:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.400+0000 7fed64a63640 1 -- 172.21.15.6:0/1975297473 wait complete. 2024-03-18T14:42:39.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.517+0000 7f5305152640 1 Processor -- start 2024-03-18T14:42:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.517+0000 7f5305152640 1 -- start start 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f5305152640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f53000afac0 0x7f530006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f5305152640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53000ce040 con 0x7f530015e030 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f5305152640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53000d0870 con 0x7f530006b700 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f5305152640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f530015e410 con 0x7f53000afac0 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f52fed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f530015e030 0x7f53000cd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50048/0 (socket says 172.21.15.6:50048) 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f52fed9d640 1 -- 172.21.15.6:0/4031271419 learned_addr learned my addr 172.21.15.6:0/4031271419 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f52fe59c640 1 --2- 172.21.15.6:0/4031271419 >> v2:172.21.15.92:3300/0 conn(0x7f53000afac0 0x7f530006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f52fdd9b640 1 -- 172.21.15.6:0/4031271419 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206102644 0 0) 0x7f52f8001110 con 0x7f530015e030 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.518+0000 7f52fdd9b640 1 -- 172.21.15.6:0/4031271419 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52ec003520 con 0x7f530015e030 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.519+0000 7f52fe59c640 1 -- 172.21.15.6:0/4031271419 >> v1:172.21.15.92:6789/0 conn(0x7f530006b700 legacy=0x7f53000cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.519+0000 7f52fdd9b640 1 -- 172.21.15.6:0/4031271419 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531676449 0 0) 0x7f53000d0870 con 0x7f530006b700 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.519+0000 7f52fe59c640 1 -- 172.21.15.6:0/4031271419 >> v1:172.21.15.6:6789/0 conn(0x7f530015e030 legacy=0x7f53000cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.519+0000 7f52fe59c640 1 -- 172.21.15.6:0/4031271419 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53000d0b50 con 0x7f53000afac0 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.519+0000 7f52fe59c640 1 --2- 172.21.15.6:0/4031271419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53000afac0 0x7f530006b1c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f52e80097a0 tx=0x7f52e802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0495c587f7a88b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.519+0000 7f52fdd9b640 1 -- 172.21.15.6:0/4031271419 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8035020 con 0x7f53000afac0 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.519+0000 7f52fdd9b640 1 -- 172.21.15.6:0/4031271419 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e8031830 con 0x7f53000afac0 2024-03-18T14:42:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f52fdd9b640 1 -- 172.21.15.6:0/4031271419 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8031c00 con 0x7f53000afac0 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f5305152640 1 -- 172.21.15.6:0/4031271419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53000afac0 msgr2=0x7f530006b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f5305152640 1 --2- 172.21.15.6:0/4031271419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53000afac0 0x7f530006b1c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f52e80097a0 tx=0x7f52e802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f5305152640 1 -- 172.21.15.6:0/4031271419 shutdown_connections 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f5305152640 1 --2- 172.21.15.6:0/4031271419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53000afac0 0x7f530006b1c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f5305152640 1 -- 172.21.15.6:0/4031271419 >> 172.21.15.6:0/4031271419 conn(0x7f5300099a50 msgr2=0x7f5300099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f5305152640 1 -- 172.21.15.6:0/4031271419 shutdown_connections 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.520+0000 7f5305152640 1 -- 172.21.15.6:0/4031271419 wait complete. 2024-03-18T14:42:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.521+0000 7f5305152640 1 Processor -- start 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.521+0000 7f5305152640 1 -- start start 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f5305152640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f530006b700 0x7f5300152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f5305152640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53000afac0 0x7f53000c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f5305152640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f530015e030 0x7f53000c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f5305152640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f53000d1630 con 0x7f53000afac0 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f5305152640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53000d11a0 con 0x7f530015e030 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f5305152640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f53000c5380 con 0x7f530006b700 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f52fe59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53000afac0 0x7f53000c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f52fe59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53000afac0 0x7f53000c6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37014/0 (socket says 172.21.15.6:37014) 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f52fe59c640 1 -- 172.21.15.6:0/209305033 learned_addr learned my addr 172.21.15.6:0/209305033 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f52fed9d640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f530006b700 0x7f5300152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.522+0000 7f52ff59e640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f530015e030 0x7f53000c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52fe59c640 1 -- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f530006b700 msgr2=0x7f5300152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52fe59c640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f530006b700 0x7f5300152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52fe59c640 1 -- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f530015e030 msgr2=0x7f53000c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52fe59c640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f530015e030 0x7f53000c6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52fe59c640 1 -- 172.21.15.6:0/209305033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53000c5660 con 0x7f53000afac0 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52fe59c640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53000afac0 0x7f53000c6710 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f52e802b910 tx=0x7f52e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52f77fe640 1 -- 172.21.15.6:0/209305033 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8040070 con 0x7f53000afac0 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52f77fe640 1 -- 172.21.15.6:0/209305033 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e803e070 con 0x7f53000afac0 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53000c5940 con 0x7f53000afac0 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f52f77fe640 1 -- 172.21.15.6:0/209305033 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8035040 con 0x7f53000afac0 2024-03-18T14:42:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.523+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53000c4650 con 0x7f53000afac0 2024-03-18T14:42:39.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.524+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53000751b0 con 0x7f53000afac0 2024-03-18T14:42:39.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.524+0000 7f52f77fe640 1 -- 172.21.15.6:0/209305033 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f52e8038020 con 0x7f53000afac0 2024-03-18T14:42:39.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.525+0000 7f52f77fe640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f52e006bbd0 0x7f52e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.525+0000 7f52f77fe640 1 -- 172.21.15.6:0/209305033 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f52e80ac570 con 0x7f53000afac0 2024-03-18T14:42:39.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.525+0000 7f52fed9d640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f52e006bbd0 0x7f52e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.525+0000 7f52fed9d640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f52e006bbd0 0x7f52e006e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f52f8005880 tx=0x7f52f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.529+0000 7f52f77fe640 1 -- 172.21.15.6:0/209305033 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f52e8031800 con 0x7f53000afac0 2024-03-18T14:42:39.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.713+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f53000ae2e0 con 0x7f53000afac0 2024-03-18T14:42:39.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.713+0000 7f52f77fe640 1 -- 172.21.15.6:0/209305033 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f52e807a6a0 con 0x7f53000afac0 2024-03-18T14:42:39.716 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f52e006bbd0 msgr2=0x7f52e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f52e006bbd0 0x7f52e006e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f52f8005880 tx=0x7f52f8009040 comp rx=0 tx=0).stop 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53000afac0 msgr2=0x7f53000c6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53000afac0 0x7f53000c6710 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f52e802b910 tx=0x7f52e80046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 shutdown_connections 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f52e006bbd0 0x7f52e006e090 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f530015e030 0x7f53000c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53000afac0 0x7f53000c6710 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 --2- 172.21.15.6:0/209305033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f530006b700 0x7f5300152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 >> 172.21.15.6:0/209305033 conn(0x7f5300099a50 msgr2=0x7f53000aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:39.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 shutdown_connections 2024-03-18T14:42:39.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.715+0000 7f5305152640 1 -- 172.21.15.6:0/209305033 wait complete. 2024-03-18T14:42:39.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:39.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.829+0000 7f3f9c6c6640 1 Processor -- start 2024-03-18T14:42:39.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.829+0000 7f3f9c6c6640 1 -- start start 2024-03-18T14:42:39.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f9c6c6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3f9406c7a0 0x7f3f940d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f9c6c6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f9406cb80 0x7f3f940b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f9c6c6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f940d1f30 con 0x7f3f940b6df0 2024-03-18T14:42:39.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f9c6c6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3f940d2210 con 0x7f3f9406cb80 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f9c6c6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f3f940d24f0 con 0x7f3f9406c7a0 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f99c61640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f9406cb80 0x7f3f940b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f9ac63640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f940b6df0 0x7f3f940b7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50060/0 (socket says 172.21.15.6:50060) 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.830+0000 7f3f99c61640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f9406cb80 0x7f3f940b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37028/0 (socket says 172.21.15.6:37028) 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.831+0000 7f3f9ac63640 1 -- 172.21.15.6:0/602056098 learned_addr learned my addr 172.21.15.6:0/602056098 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.831+0000 7f3f9a462640 1 --2- 172.21.15.6:0/602056098 >> v2:172.21.15.92:3300/0 conn(0x7f3f9406c7a0 0x7f3f940d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.831+0000 7f3f99c61640 1 -- 172.21.15.6:0/602056098 >> v2:172.21.15.92:3300/0 conn(0x7f3f9406c7a0 msgr2=0x7f3f940d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.831+0000 7f3f99c61640 1 --2- 172.21.15.6:0/602056098 >> v2:172.21.15.92:3300/0 conn(0x7f3f9406c7a0 0x7f3f940d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.831+0000 7f3f99c61640 1 -- 172.21.15.6:0/602056098 >> v1:172.21.15.6:6789/0 conn(0x7f3f940b6df0 legacy=0x7f3f940b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.831+0000 7f3f99c61640 1 -- 172.21.15.6:0/602056098 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f940d27d0 con 0x7f3f9406cb80 2024-03-18T14:42:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.831+0000 7f3f99460640 1 -- 172.21.15.6:0/602056098 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217282016 0 0) 0x7f3f940d1f30 con 0x7f3f940b6df0 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f99c61640 1 --2- 172.21.15.6:0/602056098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f9406cb80 0x7f3f940b95e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3f940d0fa0 tx=0x7f3f8402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608339e28c8f4e37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f99460640 1 -- 172.21.15.6:0/602056098 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84034070 con 0x7f3f9406cb80 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f99460640 1 -- 172.21.15.6:0/602056098 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f84030610 con 0x7f3f9406cb80 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f99460640 1 -- 172.21.15.6:0/602056098 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840309e0 con 0x7f3f9406cb80 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/602056098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f9406cb80 msgr2=0x7f3f940b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/602056098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f9406cb80 0x7f3f940b95e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3f940d0fa0 tx=0x7f3f8402b960 comp rx=0 tx=0).stop 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/602056098 shutdown_connections 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/602056098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f9406cb80 0x7f3f940b95e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/602056098 >> v2:172.21.15.92:3300/0 conn(0x7f3f9406c7a0 0x7f3f940d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/602056098 >> 172.21.15.6:0/602056098 conn(0x7f3f94072ce0 msgr2=0x7f3f940730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/602056098 shutdown_connections 2024-03-18T14:42:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.832+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/602056098 wait complete. 2024-03-18T14:42:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.833+0000 7f3f9c6c6640 1 Processor -- start 2024-03-18T14:42:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9c6c6640 1 -- start start 2024-03-18T14:42:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9c6c6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9406c7a0 0x7f3f9414a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9c6c6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f9406cb80 0x7f3f9414c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9c6c6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f940b6df0 0x7f3f94149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9c6c6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3f940b9b20 con 0x7f3f940b6df0 2024-03-18T14:42:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9c6c6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9406cfe0 con 0x7f3f9406c7a0 2024-03-18T14:42:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9c6c6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3f940b9ca0 con 0x7f3f9406cb80 2024-03-18T14:42:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9ac63640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f940b6df0 0x7f3f94149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9ac63640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f940b6df0 0x7f3f94149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37030/0 (socket says 172.21.15.6:37030) 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9ac63640 1 -- 172.21.15.6:0/2380113256 learned_addr learned my addr 172.21.15.6:0/2380113256 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f9a462640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9406c7a0 0x7f3f9414a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.834+0000 7f3f99c61640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f9406cb80 0x7f3f9414c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9ac63640 1 -- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f9406cb80 msgr2=0x7f3f9414c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9ac63640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f9406cb80 0x7f3f9414c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9ac63640 1 -- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9406c7a0 msgr2=0x7f3f9414a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9ac63640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9406c7a0 0x7f3f9414a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9ac63640 1 -- 172.21.15.6:0/2380113256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9414afc0 con 0x7f3f940b6df0 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f99c61640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f9406cb80 0x7f3f9414c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9a462640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9406c7a0 0x7f3f9414a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9ac63640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f940b6df0 0x7f3f94149f40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00c730 tx=0x7f3f8c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f8affd640 1 -- 172.21.15.6:0/2380113256 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c018070 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f8affd640 1 -- 172.21.15.6:0/2380113256 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f8c00ce40 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9414b2a0 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f8affd640 1 -- 172.21.15.6:0/2380113256 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c00a860 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.835+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9414b860 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.836+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f9404f5b0 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.836+0000 7f3f8affd640 1 -- 172.21.15.6:0/2380113256 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3f8c01b070 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.837+0000 7f3f8affd640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f7806bbd0 0x7f3f7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.837+0000 7f3f8affd640 1 -- 172.21.15.6:0/2380113256 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3f8c08a3c0 con 0x7f3f940b6df0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.837+0000 7f3f9a462640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f7806bbd0 0x7f3f7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.837+0000 7f3f9a462640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f7806bbd0 0x7f3f7806e090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009f30 tx=0x7f3f90009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:39.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:39.841+0000 7f3f8affd640 1 -- 172.21.15.6:0/2380113256 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3f8c058460 con 0x7f3f940b6df0 2024-03-18T14:42:40.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.035+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3f940cd990 con 0x7f3f940b6df0 2024-03-18T14:42:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.035+0000 7f3f8affd640 1 -- 172.21.15.6:0/2380113256 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f3f8c05c500 con 0x7f3f940b6df0 2024-03-18T14:42:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f7806bbd0 msgr2=0x7f3f7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f7806bbd0 0x7f3f7806e090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009f30 tx=0x7f3f90009190 comp rx=0 tx=0).stop 2024-03-18T14:42:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f940b6df0 msgr2=0x7f3f94149f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f940b6df0 0x7f3f94149f40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00c730 tx=0x7f3f8c00cbf0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 shutdown_connections 2024-03-18T14:42:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3f7806bbd0 0x7f3f7806e090 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f940b6df0 0x7f3f94149f40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f9406cb80 0x7f3f9414c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 --2- 172.21.15.6:0/2380113256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9406c7a0 0x7f3f9414a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 >> 172.21.15.6:0/2380113256 conn(0x7f3f94072ce0 msgr2=0x7f3f940ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 shutdown_connections 2024-03-18T14:42:40.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.037+0000 7f3f9c6c6640 1 -- 172.21.15.6:0/2380113256 wait complete. 2024-03-18T14:42:40.051 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:42:40.051 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:42:40.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.166+0000 7fbae41da640 1 Processor -- start 2024-03-18T14:42:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.166+0000 7fbae41da640 1 -- start start 2024-03-18T14:42:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae41da640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbadc0b9650 0x7fbadc0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae41da640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbadc0d0bc0 0x7fbadc0b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae41da640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbadc0d0ff0 con 0x7fbadc15e170 2024-03-18T14:42:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae41da640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbadc0b9ae0 con 0x7fbadc0b9650 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae41da640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbadc0cef50 con 0x7fbadc0d0bc0 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae1775640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbadc0b9650 0x7fbadc0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae1775640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbadc0b9650 0x7fbadc0d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37044/0 (socket says 172.21.15.6:37044) 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae1775640 1 -- 172.21.15.6:0/4071719718 learned_addr learned my addr 172.21.15.6:0/4071719718 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae1f76640 1 --1- 172.21.15.6:0/4071719718 >> v1:172.21.15.92:6789/0 conn(0x7fbadc15e170 0x7fbadc0d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48552/0 (socket says 172.21.15.6:48552) 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.167+0000 7fbae2777640 1 --2- 172.21.15.6:0/4071719718 >> v2:172.21.15.92:3300/0 conn(0x7fbadc0d0bc0 0x7fbadc0b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.168+0000 7fbae1775640 1 -- 172.21.15.6:0/4071719718 >> v2:172.21.15.92:3300/0 conn(0x7fbadc0d0bc0 msgr2=0x7fbadc0b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.168+0000 7fbae1775640 1 --2- 172.21.15.6:0/4071719718 >> v2:172.21.15.92:3300/0 conn(0x7fbadc0d0bc0 0x7fbadc0b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.168+0000 7fbae1775640 1 -- 172.21.15.6:0/4071719718 >> v1:172.21.15.92:6789/0 conn(0x7fbadc15e170 legacy=0x7fbadc0d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.168+0000 7fbae0f74640 1 -- 172.21.15.6:0/4071719718 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790574585 0 0) 0x7fbadc0d0ff0 con 0x7fbadc15e170 2024-03-18T14:42:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.168+0000 7fbae1775640 1 -- 172.21.15.6:0/4071719718 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc0cf230 con 0x7fbadc0b9650 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.168+0000 7fbae1775640 1 --2- 172.21.15.6:0/4071719718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0d0680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0c77a0 tx=0x7fbad002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2075d319ad15b677 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae0f74640 1 -- 172.21.15.6:0/4071719718 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0034070 con 0x7fbadc0b9650 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae0f74640 1 -- 172.21.15.6:0/4071719718 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad0030860 con 0x7fbadc0b9650 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae0f74640 1 -- 172.21.15.6:0/4071719718 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0030c30 con 0x7fbadc0b9650 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 -- 172.21.15.6:0/4071719718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 msgr2=0x7fbadc0d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 --2- 172.21.15.6:0/4071719718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0d0680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0c77a0 tx=0x7fbad002b9a0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 -- 172.21.15.6:0/4071719718 shutdown_connections 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 --2- 172.21.15.6:0/4071719718 >> v2:172.21.15.92:3300/0 conn(0x7fbadc0d0bc0 0x7fbadc0b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 --2- 172.21.15.6:0/4071719718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0d0680 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 -- 172.21.15.6:0/4071719718 >> 172.21.15.6:0/4071719718 conn(0x7fbadc06c900 msgr2=0x7fbadc06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 -- 172.21.15.6:0/4071719718 shutdown_connections 2024-03-18T14:42:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.169+0000 7fbae41da640 1 -- 172.21.15.6:0/4071719718 wait complete. 2024-03-18T14:42:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.170+0000 7fbae41da640 1 Processor -- start 2024-03-18T14:42:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.170+0000 7fbae41da640 1 -- start start 2024-03-18T14:42:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae41da640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae41da640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbadc0d0bc0 0x7fbadc157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae41da640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbadc15e170 0x7fbadc159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae41da640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc0d3a40 con 0x7fbadc0b9650 2024-03-18T14:42:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae41da640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc0d2e10 con 0x7fbadc15e170 2024-03-18T14:42:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae41da640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbadc0d2820 con 0x7fbadc0d0bc0 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37060/0 (socket says 172.21.15.6:37060) 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 -- 172.21.15.6:0/2925768790 learned_addr learned my addr 172.21.15.6:0/2925768790 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1775640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbadc0d0bc0 0x7fbadc157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae2777640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbadc15e170 0x7fbadc159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 -- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbadc0d0bc0 msgr2=0x7fbadc157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbadc0d0bc0 0x7fbadc157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 -- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbadc15e170 msgr2=0x7fbadc159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbadc15e170 0x7fbadc159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1f76640 1 -- 172.21.15.6:0/2925768790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc1584c0 con 0x7fbadc0b9650 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.171+0000 7fbae1775640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbadc0d0bc0 0x7fbadc157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.172+0000 7fbae1f76640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0cd850 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbad400c460 tx=0x7fbad400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.172+0000 7fbada7fc640 1 -- 172.21.15.6:0/2925768790 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad4018070 con 0x7fbadc0b9650 2024-03-18T14:42:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.172+0000 7fbada7fc640 1 -- 172.21.15.6:0/2925768790 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad4007500 con 0x7fbadc0b9650 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.172+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc1587a0 con 0x7fbadc0b9650 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.172+0000 7fbada7fc640 1 -- 172.21.15.6:0/2925768790 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad4004a20 con 0x7fbadc0b9650 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.172+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc158d60 con 0x7fbadc0b9650 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.173+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc08afe0 con 0x7fbadc0b9650 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.173+0000 7fbada7fc640 1 -- 172.21.15.6:0/2925768790 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbad401b070 con 0x7fbadc0b9650 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.174+0000 7fbada7fc640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbac806bc00 0x7fbac806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.174+0000 7fbada7fc640 1 -- 172.21.15.6:0/2925768790 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbad408a0a0 con 0x7fbadc0b9650 2024-03-18T14:42:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.174+0000 7fbae1775640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbac806bc00 0x7fbac806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.174+0000 7fbae1775640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbac806bc00 0x7fbac806e0c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbad0036000 tx=0x7fbad0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.178+0000 7fbada7fc640 1 -- 172.21.15.6:0/2925768790 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbad40580b0 con 0x7fbadc0b9650 2024-03-18T14:42:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.360+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbadc04f620 con 0x7fbadc0b9650 2024-03-18T14:42:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.361+0000 7fbada7fc640 1 -- 172.21.15.6:0/2925768790 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7fbad405c150 con 0x7fbadc0b9650 2024-03-18T14:42:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:40.364 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbac806bc00 msgr2=0x7fbac806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbac806bc00 0x7fbac806e0c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbad0036000 tx=0x7fbad0008030 comp rx=0 tx=0).stop 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 msgr2=0x7fbadc0cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0cd850 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbad400c460 tx=0x7fbad400c920 comp rx=0 tx=0).stop 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 shutdown_connections 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbac806bc00 0x7fbac806e0c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbadc15e170 0x7fbadc159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbadc0d0bc0 0x7fbadc157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 --2- 172.21.15.6:0/2925768790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbadc0b9650 0x7fbadc0cd850 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 >> 172.21.15.6:0/2925768790 conn(0x7fbadc06c900 msgr2=0x7fbadc0b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 shutdown_connections 2024-03-18T14:42:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.364+0000 7fbae41da640 1 -- 172.21.15.6:0/2925768790 wait complete. 2024-03-18T14:42:40.378 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:40.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:40.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.480+0000 7fdda760b640 1 Processor -- start 2024-03-18T14:42:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.480+0000 7fdda760b640 1 -- start start 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda760b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda00d13a0 0x7fdda0099c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda760b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdda009a180 0x7fdda00cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda760b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdda00d0300 con 0x7fdda015e150 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda760b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fdda015e530 con 0x7fdda009a180 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda760b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdda00d17e0 con 0x7fdda00d13a0 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda5ba8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdda009a180 0x7fdda00cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda53a7640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdda015e150 0x7fdda006d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50068/0 (socket says 172.21.15.6:50068) 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda5ba8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdda009a180 0x7fdda00cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37066/0 (socket says 172.21.15.6:37066) 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda53a7640 1 -- 172.21.15.6:0/1280674495 learned_addr learned my addr 172.21.15.6:0/1280674495 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda4ba6640 1 --2- 172.21.15.6:0/1280674495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda00d13a0 0x7fdda0099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda5ba8640 1 -- 172.21.15.6:0/1280674495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda00d13a0 msgr2=0x7fdda0099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda5ba8640 1 --2- 172.21.15.6:0/1280674495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda00d13a0 0x7fdda0099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda5ba8640 1 -- 172.21.15.6:0/1280674495 >> v1:172.21.15.6:6789/0 conn(0x7fdda015e150 legacy=0x7fdda006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdda5ba8640 1 -- 172.21.15.6:0/1280674495 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda00d05e0 con 0x7fdda009a180 2024-03-18T14:42:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.481+0000 7fdd97fff640 1 -- 172.21.15.6:0/1280674495 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4021381213 0 0) 0x7fdda00d0300 con 0x7fdda015e150 2024-03-18T14:42:40.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.482+0000 7fdda5ba8640 1 --2- 172.21.15.6:0/1280674495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda009a180 0x7fdda00cc1b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009d90 tx=0x7fdd9c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0f0a42c49a003f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.482+0000 7fdd97fff640 1 -- 172.21.15.6:0/1280674495 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c039070 con 0x7fdda009a180 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.482+0000 7fdd97fff640 1 -- 172.21.15.6:0/1280674495 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9c02cbc0 con 0x7fdda009a180 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.482+0000 7fdd97fff640 1 -- 172.21.15.6:0/1280674495 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c034440 con 0x7fdda009a180 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 -- 172.21.15.6:0/1280674495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda009a180 msgr2=0x7fdda00cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 --2- 172.21.15.6:0/1280674495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda009a180 0x7fdda00cc1b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009d90 tx=0x7fdd9c02be10 comp rx=0 tx=0).stop 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 -- 172.21.15.6:0/1280674495 shutdown_connections 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 --2- 172.21.15.6:0/1280674495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda009a180 0x7fdda00cc1b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 --2- 172.21.15.6:0/1280674495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda00d13a0 0x7fdda0099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 -- 172.21.15.6:0/1280674495 >> 172.21.15.6:0/1280674495 conn(0x7fdda00b3650 msgr2=0x7fdda00b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 -- 172.21.15.6:0/1280674495 shutdown_connections 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.483+0000 7fdda760b640 1 -- 172.21.15.6:0/1280674495 wait complete. 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.484+0000 7fdda760b640 1 Processor -- start 2024-03-18T14:42:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.484+0000 7fdda760b640 1 -- start start 2024-03-18T14:42:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda760b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda009a180 0x7fdda014f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda760b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda00d13a0 0x7fdda00b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda760b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdda015e150 0x7fdda00b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda760b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdda00d0ba0 con 0x7fdda00d13a0 2024-03-18T14:42:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda760b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdda009a5d0 con 0x7fdda015e150 2024-03-18T14:42:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda760b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdda00d0e80 con 0x7fdda009a180 2024-03-18T14:42:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda00d13a0 0x7fdda00b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda00d13a0 0x7fdda00b7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37078/0 (socket says 172.21.15.6:37078) 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 -- 172.21.15.6:0/4054029378 learned_addr learned my addr 172.21.15.6:0/4054029378 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda53a7640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda009a180 0x7fdda014f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda5ba8640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdda015e150 0x7fdda00b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 -- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda009a180 msgr2=0x7fdda014f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda009a180 0x7fdda014f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 -- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdda015e150 msgr2=0x7fdda00b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdda015e150 0x7fdda00b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 -- 172.21.15.6:0/4054029378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda014f5a0 con 0x7fdda00d13a0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.485+0000 7fdda4ba6640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda00d13a0 0x7fdda00b7030 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e870 tx=0x7fdd9000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.486+0000 7fdd95ffb640 1 -- 172.21.15.6:0/4054029378 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90017070 con 0x7fdda00d13a0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.486+0000 7fdd95ffb640 1 -- 172.21.15.6:0/4054029378 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9000c940 con 0x7fdda00d13a0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.486+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda00c8380 con 0x7fdda00d13a0 2024-03-18T14:42:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.486+0000 7fdd95ffb640 1 -- 172.21.15.6:0/4054029378 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9000cd30 con 0x7fdda00d13a0 2024-03-18T14:42:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.486+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda00c7ae0 con 0x7fdda00d13a0 2024-03-18T14:42:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.487+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda00c9d20 con 0x7fdda00d13a0 2024-03-18T14:42:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.487+0000 7fdd95ffb640 1 -- 172.21.15.6:0/4054029378 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdd9001a070 con 0x7fdda00d13a0 2024-03-18T14:42:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.487+0000 7fdd95ffb640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8406bbd0 0x7fdd8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.488+0000 7fdd95ffb640 1 -- 172.21.15.6:0/4054029378 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdd9008b060 con 0x7fdda00d13a0 2024-03-18T14:42:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.488+0000 7fdda53a7640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8406bbd0 0x7fdd8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.488+0000 7fdda53a7640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8406bbd0 0x7fdd8406e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd98007fd0 tx=0x7fdd98007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.492+0000 7fdd95ffb640 1 -- 172.21.15.6:0/4054029378 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdda00c9d20 con 0x7fdda00d13a0 2024-03-18T14:42:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.672+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdda00751b0 con 0x7fdda00d13a0 2024-03-18T14:42:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.673+0000 7fdd95ffb640 1 -- 172.21.15.6:0/4054029378 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fdda014f5a0 con 0x7fdda00d13a0 2024-03-18T14:42:40.675 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8406bbd0 msgr2=0x7fdd8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8406bbd0 0x7fdd8406e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd98007fd0 tx=0x7fdd98007de0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda00d13a0 msgr2=0x7fdda00b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda00d13a0 0x7fdda00b7030 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e870 tx=0x7fdd9000ed30 comp rx=0 tx=0).stop 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 shutdown_connections 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd8406bbd0 0x7fdd8406e090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdda015e150 0x7fdda00b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdda00d13a0 0x7fdda00b7030 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 --2- 172.21.15.6:0/4054029378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdda009a180 0x7fdda014f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 >> 172.21.15.6:0/4054029378 conn(0x7fdda00b3650 msgr2=0x7fdda00af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 shutdown_connections 2024-03-18T14:42:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.675+0000 7fdda760b640 1 -- 172.21.15.6:0/4054029378 wait complete. 2024-03-18T14:42:40.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:40.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.790+0000 7f9f5d6e9640 1 Processor -- start 2024-03-18T14:42:40.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.790+0000 7f9f5d6e9640 1 -- start start 2024-03-18T14:42:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f5d6e9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f580b0ea0 0x7f9f58165af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f5d6e9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f5815f6c0 con 0x7f9f58143ed0 2024-03-18T14:42:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f5d6e9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f5815e900 con 0x7f9f581443c0 2024-03-18T14:42:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f5d6e9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9f580cf7d0 con 0x7f9f580b0ea0 2024-03-18T14:42:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f56ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9f58143ed0 0x7f9f580cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50084/0 (socket says 172.21.15.6:50084) 2024-03-18T14:42:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f56ffd640 1 -- 172.21.15.6:0/1956966540 learned_addr learned my addr 172.21.15.6:0/1956966540 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f577fe640 1 --2- 172.21.15.6:0/1956966540 >> v2:172.21.15.92:3300/0 conn(0x7f9f580b0ea0 0x7f9f58165af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f55ffb640 1 -- 172.21.15.6:0/1956966540 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896143827 0 0) 0x7f9f5815f6c0 con 0x7f9f58143ed0 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f55ffb640 1 -- 172.21.15.6:0/1956966540 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f44003520 con 0x7f9f58143ed0 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f55ffb640 1 -- 172.21.15.6:0/1956966540 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906826521 0 0) 0x7f9f5815e900 con 0x7f9f581443c0 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f55ffb640 1 -- 172.21.15.6:0/1956966540 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f5815f6c0 con 0x7f9f581443c0 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f577fe640 1 -- 172.21.15.6:0/1956966540 >> v1:172.21.15.92:6789/0 conn(0x7f9f581443c0 legacy=0x7f9f580b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f577fe640 1 -- 172.21.15.6:0/1956966540 >> v1:172.21.15.6:6789/0 conn(0x7f9f58143ed0 legacy=0x7f9f580cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.791+0000 7f9f577fe640 1 -- 172.21.15.6:0/1956966540 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f5815ebe0 con 0x7f9f580b0ea0 2024-03-18T14:42:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.792+0000 7f9f577fe640 1 --2- 172.21.15.6:0/1956966540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 0x7f9f58165af0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009d90 tx=0x7f9f4802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e92db4194a629b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.792+0000 7f9f55ffb640 1 -- 172.21.15.6:0/1956966540 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48039070 con 0x7f9f580b0ea0 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.792+0000 7f9f55ffb640 1 -- 172.21.15.6:0/1956966540 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4802cd00 con 0x7f9f580b0ea0 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.792+0000 7f9f55ffb640 1 -- 172.21.15.6:0/1956966540 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48034660 con 0x7f9f580b0ea0 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.793+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1956966540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 msgr2=0x7f9f58165af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.793+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1956966540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 0x7f9f58165af0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009d90 tx=0x7f9f4802bf50 comp rx=0 tx=0).stop 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.793+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1956966540 shutdown_connections 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.793+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1956966540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 0x7f9f58165af0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.793+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1956966540 >> 172.21.15.6:0/1956966540 conn(0x7f9f5806b1c0 msgr2=0x7f9f5806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.793+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1956966540 shutdown_connections 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.793+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1956966540 wait complete. 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.794+0000 7f9f5d6e9640 1 Processor -- start 2024-03-18T14:42:40.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.794+0000 7f9f5d6e9640 1 -- start start 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f5d6e9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 0x7f9f5815a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f5d6e9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f58143ed0 0x7f9f580a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f5d6e9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f581443c0 0x7f9f580ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f5d6e9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5815f230 con 0x7f9f581443c0 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f5d6e9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9f580cf590 con 0x7f9f580b0ea0 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f5d6e9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9f580b4880 con 0x7f9f58143ed0 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f581443c0 0x7f9f580ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f581443c0 0x7f9f580ab020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37080/0 (socket says 172.21.15.6:37080) 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 -- 172.21.15.6:0/1438686995 learned_addr learned my addr 172.21.15.6:0/1438686995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:40.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f56ffd640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 0x7f9f5815a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f567fc640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f58143ed0 0x7f9f580a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 -- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f58143ed0 msgr2=0x7f9f580a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f58143ed0 0x7f9f580a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 -- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 msgr2=0x7f9f5815a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 0x7f9f5815a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.795+0000 7f9f577fe640 1 -- 172.21.15.6:0/1438686995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f58140940 con 0x7f9f581443c0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.796+0000 7f9f577fe640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f581443c0 0x7f9f580ab020 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009d90 tx=0x7f9f48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.796+0000 7f9f4f7fe640 1 -- 172.21.15.6:0/1438686995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48039050 con 0x7f9f581443c0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.796+0000 7f9f4f7fe640 1 -- 172.21.15.6:0/1438686995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f48045070 con 0x7f9f581443c0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.796+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f58140c20 con 0x7f9f581443c0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.796+0000 7f9f4f7fe640 1 -- 172.21.15.6:0/1438686995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48009450 con 0x7f9f581443c0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.796+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f581411e0 con 0x7f9f581443c0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.797+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f580751b0 con 0x7f9f581443c0 2024-03-18T14:42:40.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.797+0000 7f9f4f7fe640 1 -- 172.21.15.6:0/1438686995 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9f4803e850 con 0x7f9f581443c0 2024-03-18T14:42:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.798+0000 7f9f4f7fe640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9f3806bbd0 0x7f9f3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.798+0000 7f9f4f7fe640 1 -- 172.21.15.6:0/1438686995 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9f480adb00 con 0x7f9f581443c0 2024-03-18T14:42:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.798+0000 7f9f56ffd640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9f3806bbd0 0x7f9f3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.798+0000 7f9f56ffd640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9f3806bbd0 0x7f9f3806e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9f50000f60 tx=0x7f9f50007d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:40.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.802+0000 7f9f4f7fe640 1 -- 172.21.15.6:0/1438686995 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9f4807bcc0 con 0x7f9f581443c0 2024-03-18T14:42:40.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.983+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9f580b3260 con 0x7f9f581443c0 2024-03-18T14:42:40.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.983+0000 7f9f4f7fe640 1 -- 172.21.15.6:0/1438686995 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f9f4807fd60 con 0x7f9f581443c0 2024-03-18T14:42:40.986 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9f3806bbd0 msgr2=0x7f9f3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9f3806bbd0 0x7f9f3806e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9f50000f60 tx=0x7f9f50007d70 comp rx=0 tx=0).stop 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f581443c0 msgr2=0x7f9f580ab020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f581443c0 0x7f9f580ab020 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009d90 tx=0x7f9f48004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 shutdown_connections 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9f3806bbd0 0x7f9f3806e090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f581443c0 0x7f9f580ab020 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f58143ed0 0x7f9f580a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 --2- 172.21.15.6:0/1438686995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f580b0ea0 0x7f9f5815a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 >> 172.21.15.6:0/1438686995 conn(0x7f9f5806b1c0 msgr2=0x7f9f580b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:40.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 shutdown_connections 2024-03-18T14:42:40.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:40.985+0000 7f9f5d6e9640 1 -- 172.21.15.6:0/1438686995 wait complete. 2024-03-18T14:42:40.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:41.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.100+0000 7fc968bf3640 1 Processor -- start 2024-03-18T14:42:41.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.100+0000 7fc968bf3640 1 -- start start 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc968bf3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc9640cf6a0 0x7fc9640ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc968bf3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640cb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc968bf3640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9640b2280 con 0x7fc9640b1ea0 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc968bf3640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc9640ceb60 con 0x7fc9640cf6a0 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc968bf3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc96415b3e0 con 0x7fc9640d14d0 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc96259c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc9640b1ea0 0x7fc9640cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48566/0 (socket says 172.21.15.6:48566) 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc95bfff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc9640cf6a0 0x7fc9640ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc96259c640 1 -- 172.21.15.6:0/3100514225 learned_addr learned my addr 172.21.15.6:0/3100514225 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc962d9d640 1 --2- 172.21.15.6:0/3100514225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc962d9d640 1 -- 172.21.15.6:0/3100514225 >> v2:172.21.15.92:3300/0 conn(0x7fc9640cf6a0 msgr2=0x7fc9640ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:41.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc962d9d640 1 --2- 172.21.15.6:0/3100514225 >> v2:172.21.15.92:3300/0 conn(0x7fc9640cf6a0 0x7fc9640ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc962d9d640 1 -- 172.21.15.6:0/3100514225 >> v1:172.21.15.92:6789/0 conn(0x7fc9640b1ea0 legacy=0x7fc9640cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc961d9b640 1 -- 172.21.15.6:0/3100514225 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488970216 0 0) 0x7fc9640b2280 con 0x7fc9640b1ea0 2024-03-18T14:42:41.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc962d9d640 1 -- 172.21.15.6:0/3100514225 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96415b6c0 con 0x7fc9640d14d0 2024-03-18T14:42:41.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.101+0000 7fc95bfff640 1 --2- 172.21.15.6:0/3100514225 >> v2:172.21.15.92:3300/0 conn(0x7fc9640cf6a0 0x7fc9640ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:41.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.102+0000 7fc962d9d640 1 --2- 172.21.15.6:0/3100514225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640cb530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009d90 tx=0x7fc95c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b1e809072af796 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.102+0000 7fc961d9b640 1 -- 172.21.15.6:0/3100514225 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c039070 con 0x7fc9640d14d0 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.102+0000 7fc961d9b640 1 -- 172.21.15.6:0/3100514225 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc95c02cc50 con 0x7fc9640d14d0 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.102+0000 7fc961d9b640 1 -- 172.21.15.6:0/3100514225 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c034440 con 0x7fc9640d14d0 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 -- 172.21.15.6:0/3100514225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 msgr2=0x7fc9640cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 --2- 172.21.15.6:0/3100514225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640cb530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009d90 tx=0x7fc95c02bea0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 -- 172.21.15.6:0/3100514225 shutdown_connections 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 --2- 172.21.15.6:0/3100514225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640cb530 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 --2- 172.21.15.6:0/3100514225 >> v2:172.21.15.92:3300/0 conn(0x7fc9640cf6a0 0x7fc9640ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 -- 172.21.15.6:0/3100514225 >> 172.21.15.6:0/3100514225 conn(0x7fc9640d09b0 msgr2=0x7fc9640d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 -- 172.21.15.6:0/3100514225 shutdown_connections 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.103+0000 7fc968bf3640 1 -- 172.21.15.6:0/3100514225 wait complete. 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.104+0000 7fc968bf3640 1 Processor -- start 2024-03-18T14:42:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.104+0000 7fc968bf3640 1 -- start start 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc968bf3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9640b1ea0 0x7fc9641f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc968bf3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9640cf6a0 0x7fc9640af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc968bf3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc968bf3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc96415b0d0 con 0x7fc9640cf6a0 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc968bf3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc9640cdcf0 con 0x7fc9640b1ea0 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc968bf3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc96415bec0 con 0x7fc9640d14d0 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9640cf6a0 0x7fc9640af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc96259c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9640b1ea0 0x7fc9641f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc962d9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9640cf6a0 0x7fc9640af060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37090/0 (socket says 172.21.15.6:37090) 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 -- 172.21.15.6:0/800870175 learned_addr learned my addr 172.21.15.6:0/800870175 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 -- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 msgr2=0x7fc9640af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 -- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9640b1ea0 msgr2=0x7fc9641f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9640b1ea0 0x7fc9641f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 -- 172.21.15.6:0/800870175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9641f1cc0 con 0x7fc9640cf6a0 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc96259c640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9640b1ea0 0x7fc9641f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95bfff640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9640cf6a0 0x7fc9640af060 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00ebe0 tx=0x7fc94c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95affd640 1 -- 172.21.15.6:0/800870175 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c017070 con 0x7fc9640cf6a0 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.105+0000 7fc95affd640 1 -- 172.21.15.6:0/800870175 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc94c00cc30 con 0x7fc9640cf6a0 2024-03-18T14:42:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.106+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96409dc00 con 0x7fc9640cf6a0 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.106+0000 7fc95affd640 1 -- 172.21.15.6:0/800870175 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c013470 con 0x7fc9640cf6a0 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.106+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9640b1660 con 0x7fc9640cf6a0 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.107+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc964071270 con 0x7fc9640cf6a0 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.107+0000 7fc95affd640 1 -- 172.21.15.6:0/800870175 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc94c01a070 con 0x7fc9640cf6a0 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.107+0000 7fc95affd640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc94406bbd0 0x7fc94406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.107+0000 7fc95affd640 1 -- 172.21.15.6:0/800870175 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc94c08ad30 con 0x7fc9640cf6a0 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.108+0000 7fc96259c640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc94406bbd0 0x7fc94406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.108+0000 7fc96259c640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc94406bbd0 0x7fc94406e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc954000f60 tx=0x7fc954009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.112+0000 7fc95affd640 1 -- 172.21.15.6:0/800870175 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc964071270 con 0x7fc9640cf6a0 2024-03-18T14:42:41.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.293+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc964068210 con 0x7fc9640cf6a0 2024-03-18T14:42:41.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.293+0000 7fc95affd640 1 -- 172.21.15.6:0/800870175 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc964068210 con 0x7fc9640cf6a0 2024-03-18T14:42:41.296 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc94406bbd0 msgr2=0x7fc94406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc94406bbd0 0x7fc94406e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc954000f60 tx=0x7fc954009040 comp rx=0 tx=0).stop 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9640cf6a0 msgr2=0x7fc9640af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9640cf6a0 0x7fc9640af060 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00ebe0 tx=0x7fc94c00c520 comp rx=0 tx=0).stop 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 shutdown_connections 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc94406bbd0 0x7fc94406e090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9640d14d0 0x7fc9640af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9640cf6a0 0x7fc9640af060 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 --2- 172.21.15.6:0/800870175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc9640b1ea0 0x7fc9641f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 >> 172.21.15.6:0/800870175 conn(0x7fc9640d09b0 msgr2=0x7fc9640d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:41.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 shutdown_connections 2024-03-18T14:42:41.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.295+0000 7fc968bf3640 1 -- 172.21.15.6:0/800870175 wait complete. 2024-03-18T14:42:41.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:41.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.410+0000 7f445723d640 1 Processor -- start 2024-03-18T14:42:41.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.410+0000 7f445723d640 1 -- start start 2024-03-18T14:42:41.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.411+0000 7f445723d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.411+0000 7f445723d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44500cfac0 con 0x7f44500d0730 2024-03-18T14:42:41.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.411+0000 7f445723d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44500cccc0 con 0x7f44500ce9d0 2024-03-18T14:42:41.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.411+0000 7f445723d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f44500cd8d0 con 0x7f44500b2470 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.411+0000 7f444ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44500d0730 0x7f445015b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50086/0 (socket says 172.21.15.6:50086) 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.411+0000 7f444ffff640 1 -- 172.21.15.6:0/1393327314 learned_addr learned my addr 172.21.15.6:0/1393327314 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.411+0000 7f44557da640 1 --2- 172.21.15.6:0/1393327314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f444f7fe640 1 -- 172.21.15.6:0/1393327314 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 306124541 0 0) 0x7f44500cfac0 con 0x7f44500d0730 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f444f7fe640 1 -- 172.21.15.6:0/1393327314 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4438003520 con 0x7f44500d0730 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f444f7fe640 1 -- 172.21.15.6:0/1393327314 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 184034700 0 0) 0x7f44500cccc0 con 0x7f44500ce9d0 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f444f7fe640 1 -- 172.21.15.6:0/1393327314 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44500cfac0 con 0x7f44500ce9d0 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f44557da640 1 -- 172.21.15.6:0/1393327314 >> v1:172.21.15.92:6789/0 conn(0x7f44500ce9d0 legacy=0x7f44500d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f44557da640 1 -- 172.21.15.6:0/1393327314 >> v1:172.21.15.6:6789/0 conn(0x7f44500d0730 legacy=0x7f445015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f44557da640 1 -- 172.21.15.6:0/1393327314 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44500ccfa0 con 0x7f44500b2470 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.412+0000 7f44557da640 1 --2- 172.21.15.6:0/1393327314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500cc650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44400097d0 tx=0x7f444002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24c7053679330e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f444f7fe640 1 -- 172.21.15.6:0/1393327314 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440035020 con 0x7f44500b2470 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f444f7fe640 1 -- 172.21.15.6:0/1393327314 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4440031800 con 0x7f44500b2470 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f444f7fe640 1 -- 172.21.15.6:0/1393327314 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440031bd0 con 0x7f44500b2470 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f445723d640 1 -- 172.21.15.6:0/1393327314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 msgr2=0x7f44500cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f445723d640 1 --2- 172.21.15.6:0/1393327314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500cc650 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44400097d0 tx=0x7f444002b8a0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f445723d640 1 -- 172.21.15.6:0/1393327314 shutdown_connections 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f445723d640 1 --2- 172.21.15.6:0/1393327314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500cc650 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f445723d640 1 -- 172.21.15.6:0/1393327314 >> 172.21.15.6:0/1393327314 conn(0x7f445008b0c0 msgr2=0x7f44500d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f445723d640 1 -- 172.21.15.6:0/1393327314 shutdown_connections 2024-03-18T14:42:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.413+0000 7f445723d640 1 -- 172.21.15.6:0/1393327314 wait complete. 2024-03-18T14:42:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.414+0000 7f445723d640 1 Processor -- start 2024-03-18T14:42:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.414+0000 7f445723d640 1 -- start start 2024-03-18T14:42:41.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f445723d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f445723d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44500ce9d0 0x7f44500b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f445723d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44500d0730 0x7f445006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f445723d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f44500cedb0 con 0x7f44500d0730 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f445723d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f44500cd560 con 0x7f44500ce9d0 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f445723d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f44500d2510 con 0x7f44500b2470 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f44557da640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44500d0730 0x7f445006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f44557da640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44500d0730 0x7f445006d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37102/0 (socket says 172.21.15.6:37102) 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f44557da640 1 -- 172.21.15.6:0/1040409226 learned_addr learned my addr 172.21.15.6:0/1040409226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f4454fd9640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.415+0000 7f444ffff640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44500ce9d0 0x7f44500b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f44557da640 1 -- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 msgr2=0x7f44500c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f44557da640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f44557da640 1 -- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44500ce9d0 msgr2=0x7f44500b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f44557da640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44500ce9d0 0x7f44500b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f44557da640 1 -- 172.21.15.6:0/1040409226 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445006c450 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f44557da640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44500d0730 0x7f445006d4f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f44400098f0 tx=0x7f44400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f444d7fa640 1 -- 172.21.15.6:0/1040409226 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440035040 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f444d7fa640 1 -- 172.21.15.6:0/1040409226 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444003ac80 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445006c730 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f444d7fa640 1 -- 172.21.15.6:0/1040409226 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440046480 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.416+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445006ccf0 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.417+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44500751b0 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.417+0000 7f444d7fa640 1 -- 172.21.15.6:0/1040409226 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4440038020 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.418+0000 7f444d7fa640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f443c06bbd0 0x7f443c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.418+0000 7f444d7fa640 1 -- 172.21.15.6:0/1040409226 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f44400acec0 con 0x7f44500d0730 2024-03-18T14:42:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.418+0000 7f4454fd9640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f443c06bbd0 0x7f443c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.418+0000 7f4454fd9640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f443c06bbd0 0x7f443c06e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4448007fd0 tx=0x7f4448009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.422+0000 7f444d7fa640 1 -- 172.21.15.6:0/1040409226 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f444007af60 con 0x7f44500d0730 2024-03-18T14:42:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.604+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f44500cfe00 con 0x7f44500d0730 2024-03-18T14:42:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.604+0000 7f444d7fa640 1 -- 172.21.15.6:0/1040409226 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f444007f000 con 0x7f44500d0730 2024-03-18T14:42:41.607 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f443c06bbd0 msgr2=0x7f443c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f443c06bbd0 0x7f443c06e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4448007fd0 tx=0x7f4448009040 comp rx=0 tx=0).stop 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44500d0730 msgr2=0x7f445006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44500d0730 0x7f445006d4f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f44400098f0 tx=0x7f44400046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 shutdown_connections 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f443c06bbd0 0x7f443c06e090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44500d0730 0x7f445006d4f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44500ce9d0 0x7f44500b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 --2- 172.21.15.6:0/1040409226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44500b2470 0x7f44500c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 >> 172.21.15.6:0/1040409226 conn(0x7f445008b0c0 msgr2=0x7f44500d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 shutdown_connections 2024-03-18T14:42:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.606+0000 7f445723d640 1 -- 172.21.15.6:0/1040409226 wait complete. 2024-03-18T14:42:41.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:42:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.719+0000 7f1d181a9640 1 Processor -- start 2024-03-18T14:42:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.719+0000 7f1d181a9640 1 -- start start 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d181a9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d100d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d181a9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1d1006c4e0 0x7f1d100b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d181a9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d1006ca50 con 0x7f1d100b1f00 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d181a9640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1d1006ce10 con 0x7f1d1006c4e0 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d181a9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1d1006d0f0 con 0x7f1d1015e010 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d15744640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1d1006c4e0 0x7f1d100b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d16746640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1d100b1f00 0x7f1d100ce5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50102/0 (socket says 172.21.15.6:50102) 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d16746640 1 -- 172.21.15.6:0/423749340 learned_addr learned my addr 172.21.15.6:0/423749340 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d15f45640 1 --2- 172.21.15.6:0/423749340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d100d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d15744640 1 -- 172.21.15.6:0/423749340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 msgr2=0x7f1d100d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d15744640 1 --2- 172.21.15.6:0/423749340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d100d0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d15744640 1 -- 172.21.15.6:0/423749340 >> v1:172.21.15.6:6789/0 conn(0x7f1d100b1f00 legacy=0x7f1d100ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d15744640 1 -- 172.21.15.6:0/423749340 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d100b34f0 con 0x7f1d1006c4e0 2024-03-18T14:42:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d14f43640 1 -- 172.21.15.6:0/423749340 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523829364 0 0) 0x7f1d1006ca50 con 0x7f1d100b1f00 2024-03-18T14:42:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.720+0000 7f1d15f45640 1 --2- 172.21.15.6:0/423749340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d100d0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.721+0000 7f1d15744640 1 --2- 172.21.15.6:0/423749340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b19c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1d040097a0 tx=0x7f1d0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1c0319d3b2a8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.721+0000 7f1d14f43640 1 -- 172.21.15.6:0/423749340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d040315a0 con 0x7f1d1006c4e0 2024-03-18T14:42:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.721+0000 7f1d14f43640 1 -- 172.21.15.6:0/423749340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d04037070 con 0x7f1d1006c4e0 2024-03-18T14:42:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.721+0000 7f1d14f43640 1 -- 172.21.15.6:0/423749340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04031960 con 0x7f1d1006c4e0 2024-03-18T14:42:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 -- 172.21.15.6:0/423749340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 msgr2=0x7f1d100b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 --2- 172.21.15.6:0/423749340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b19c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1d040097a0 tx=0x7f1d0402b840 comp rx=0 tx=0).stop 2024-03-18T14:42:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 -- 172.21.15.6:0/423749340 shutdown_connections 2024-03-18T14:42:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 --2- 172.21.15.6:0/423749340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b19c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 --2- 172.21.15.6:0/423749340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d100d0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 -- 172.21.15.6:0/423749340 >> 172.21.15.6:0/423749340 conn(0x7f1d1008b320 msgr2=0x7f1d1008b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 -- 172.21.15.6:0/423749340 shutdown_connections 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.722+0000 7f1d181a9640 1 -- 172.21.15.6:0/423749340 wait complete. 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.723+0000 7f1d181a9640 1 Processor -- start 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.723+0000 7f1d181a9640 1 -- start start 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d181a9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d181a9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d100b1f00 0x7f1d100b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d181a9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d1014e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d181a9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1015e570 con 0x7f1d1006c4e0 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d181a9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1015e3f0 con 0x7f1d100b1f00 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d181a9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1d100b2390 con 0x7f1d1015e010 2024-03-18T14:42:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37128/0 (socket says 172.21.15.6:37128) 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 -- 172.21.15.6:0/2152118294 learned_addr learned my addr 172.21.15.6:0/2152118294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d16746640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d1014e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15744640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d100b1f00 0x7f1d100b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 -- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 msgr2=0x7f1d1014e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d1014e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 -- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d100b1f00 msgr2=0x7f1d100b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d100b1f00 0x7f1d100b76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 -- 172.21.15.6:0/2152118294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d100b7c10 con 0x7f1d1006c4e0 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d15f45640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b4910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800e960 tx=0x7f1d0800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d0e7fc640 1 -- 172.21.15.6:0/2152118294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08017070 con 0x7f1d1006c4e0 2024-03-18T14:42:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.724+0000 7f1d0e7fc640 1 -- 172.21.15.6:0/2152118294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0800ca00 con 0x7f1d1006c4e0 2024-03-18T14:42:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.725+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d100c7290 con 0x7f1d1006c4e0 2024-03-18T14:42:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.725+0000 7f1d0e7fc640 1 -- 172.21.15.6:0/2152118294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0800cdd0 con 0x7f1d1006c4e0 2024-03-18T14:42:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.725+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d100c7850 con 0x7f1d1006c4e0 2024-03-18T14:42:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.725+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d1004f5b0 con 0x7f1d1006c4e0 2024-03-18T14:42:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.726+0000 7f1d0e7fc640 1 -- 172.21.15.6:0/2152118294 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1d0801a070 con 0x7f1d1006c4e0 2024-03-18T14:42:41.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.726+0000 7f1d0e7fc640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1cfc06bbd0 0x7f1cfc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.726+0000 7f1d0e7fc640 1 -- 172.21.15.6:0/2152118294 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1d0808acc0 con 0x7f1d1006c4e0 2024-03-18T14:42:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.726+0000 7f1d15744640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1cfc06bbd0 0x7f1cfc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.727+0000 7f1d15744640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1cfc06bbd0 0x7f1cfc06e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1d04036000 tx=0x7f1d04008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:41.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.731+0000 7f1d0e7fc640 1 -- 172.21.15.6:0/2152118294 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1d08058d60 con 0x7f1d1006c4e0 2024-03-18T14:42:41.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.925+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1d100afb20 con 0x7f1d1006c4e0 2024-03-18T14:42:41.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.925+0000 7f1d0e7fc640 1 -- 172.21.15.6:0/2152118294 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f1d0805ce00 con 0x7f1d1006c4e0 2024-03-18T14:42:41.927 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:42:41.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1cfc06bbd0 msgr2=0x7f1cfc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1cfc06bbd0 0x7f1cfc06e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1d04036000 tx=0x7f1d04008030 comp rx=0 tx=0).stop 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 msgr2=0x7f1d100b4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b4910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800e960 tx=0x7f1d0800ee20 comp rx=0 tx=0).stop 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 shutdown_connections 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1cfc06bbd0 0x7f1cfc06e090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d1015e010 0x7f1d1014e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d100b1f00 0x7f1d100b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 --2- 172.21.15.6:0/2152118294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d1006c4e0 0x7f1d100b4910 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 >> 172.21.15.6:0/2152118294 conn(0x7f1d1008b320 msgr2=0x7f1d1015d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 shutdown_connections 2024-03-18T14:42:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:41.927+0000 7f1d181a9640 1 -- 172.21.15.6:0/2152118294 wait complete. 2024-03-18T14:42:41.940 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi092.front.sepia.ceph.com... 2024-03-18T14:42:41.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:41.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-18T14:42:41.969 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:41.970 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:42:41.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:42:41.970 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T14:42:42.024 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T14:42:42.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:42.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.123+0000 7f0a77b97640 1 Processor -- start 2024-03-18T14:42:42.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.123+0000 7f0a77b97640 1 -- start start 2024-03-18T14:42:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a77b97640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0a7015bd40 0x7f0a700d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a77b97640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700d1d90 0x7f0a700cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a77b97640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a700cfcd0 0x7f0a700ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a77b97640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0a700cc870 con 0x7f0a700cfcd0 2024-03-18T14:42:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a77b97640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0a700d0500 con 0x7f0a7015bd40 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a77b97640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0a700d07e0 con 0x7f0a700d1d90 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a76134640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a700cfcd0 0x7f0a700ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a76134640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a700cfcd0 0x7f0a700ce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37132/0 (socket says 172.21.15.6:37132) 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a76134640 1 -- 172.21.15.6:0/1590858886 learned_addr learned my addr 172.21.15.6:0/1590858886 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a75132640 1 --2- 172.21.15.6:0/1590858886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700d1d90 0x7f0a700cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.124+0000 7f0a75933640 1 --2- 172.21.15.6:0/1590858886 >> v2:172.21.15.92:3300/0 conn(0x7f0a7015bd40 0x7f0a700d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.125+0000 7f0a76134640 1 -- 172.21.15.6:0/1590858886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700d1d90 msgr2=0x7f0a700cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.125+0000 7f0a76134640 1 --2- 172.21.15.6:0/1590858886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700d1d90 0x7f0a700cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.125+0000 7f0a76134640 1 -- 172.21.15.6:0/1590858886 >> v2:172.21.15.92:3300/0 conn(0x7f0a7015bd40 msgr2=0x7f0a700d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.125+0000 7f0a76134640 1 --2- 172.21.15.6:0/1590858886 >> v2:172.21.15.92:3300/0 conn(0x7f0a7015bd40 0x7f0a700d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.125+0000 7f0a76134640 1 -- 172.21.15.6:0/1590858886 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a700ccb10 con 0x7f0a700cfcd0 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.125+0000 7f0a76134640 1 --2- 172.21.15.6:0/1590858886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700cfcd0 0x7f0a700ce540 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0a7008bae0 tx=0x7f0a6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647fbf3d6bcb2df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a74931640 1 -- 172.21.15.6:0/1590858886 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68035020 con 0x7f0a700cfcd0 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a74931640 1 -- 172.21.15.6:0/1590858886 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a68031870 con 0x7f0a700cfcd0 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a74931640 1 -- 172.21.15.6:0/1590858886 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68031c40 con 0x7f0a700cfcd0 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 -- 172.21.15.6:0/1590858886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700cfcd0 msgr2=0x7f0a700ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1590858886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700cfcd0 0x7f0a700ce540 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0a7008bae0 tx=0x7f0a6802b910 comp rx=0 tx=0).stop 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 -- 172.21.15.6:0/1590858886 shutdown_connections 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1590858886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700cfcd0 0x7f0a700ce540 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1590858886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700d1d90 0x7f0a700cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1590858886 >> v2:172.21.15.92:3300/0 conn(0x7f0a7015bd40 0x7f0a700d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 -- 172.21.15.6:0/1590858886 >> 172.21.15.6:0/1590858886 conn(0x7f0a7009ae20 msgr2=0x7f0a7008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 -- 172.21.15.6:0/1590858886 shutdown_connections 2024-03-18T14:42:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.126+0000 7f0a77b97640 1 -- 172.21.15.6:0/1590858886 wait complete. 2024-03-18T14:42:42.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.127+0000 7f0a77b97640 1 Processor -- start 2024-03-18T14:42:42.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.127+0000 7f0a77b97640 1 -- start start 2024-03-18T14:42:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a77b97640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700cfcd0 0x7f0a700af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a77b97640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700d1d90 0x7f0a7015d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a77b97640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a7015bd40 0x7f0a70157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a77b97640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0a700d0c40 con 0x7f0a700d1d90 2024-03-18T14:42:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a77b97640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0a700d0ac0 con 0x7f0a7015bd40 2024-03-18T14:42:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a77b97640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0a700d0160 con 0x7f0a700cfcd0 2024-03-18T14:42:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a75132640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700d1d90 0x7f0a7015d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a75132640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700d1d90 0x7f0a7015d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37138/0 (socket says 172.21.15.6:37138) 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a75132640 1 -- 172.21.15.6:0/1217211810 learned_addr learned my addr 172.21.15.6:0/1217211810 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a75933640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700cfcd0 0x7f0a700af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.128+0000 7f0a76134640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a7015bd40 0x7f0a70157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a75933640 1 -- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a7015bd40 msgr2=0x7f0a70157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a75933640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a7015bd40 0x7f0a70157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a75933640 1 -- 172.21.15.6:0/1217211810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700d1d90 msgr2=0x7f0a7015d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a75933640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700d1d90 0x7f0a7015d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a75933640 1 -- 172.21.15.6:0/1217211810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a700af7c0 con 0x7f0a700cfcd0 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a75132640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700d1d90 0x7f0a7015d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a75933640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700cfcd0 0x7f0a700af1e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000ebc0 tx=0x7f0a6000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a6e7fc640 1 -- 172.21.15.6:0/1217211810 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60018020 con 0x7f0a700cfcd0 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a6e7fc640 1 -- 172.21.15.6:0/1217211810 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6000cda0 con 0x7f0a700cfcd0 2024-03-18T14:42:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a700afaa0 con 0x7f0a700cfcd0 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a6e7fc640 1 -- 172.21.15.6:0/1217211810 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a600146c0 con 0x7f0a700cfcd0 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.129+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a700b5e30 con 0x7f0a700cfcd0 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.130+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7009fa90 con 0x7f0a700cfcd0 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.130+0000 7f0a6e7fc640 1 -- 172.21.15.6:0/1217211810 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0a60005d10 con 0x7f0a700cfcd0 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.131+0000 7f0a6e7fc640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a5c06bbd0 0x7f0a5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.131+0000 7f0a6e7fc640 1 -- 172.21.15.6:0/1217211810 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0a6008b9b0 con 0x7f0a700cfcd0 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.131+0000 7f0a75132640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a5c06bbd0 0x7f0a5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.131+0000 7f0a75132640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a5c06bbd0 0x7f0a5c06e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0a700b0cb0 tx=0x7f0a64009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.136+0000 7f0a6e7fc640 1 -- 172.21.15.6:0/1217211810 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0a60022030 con 0x7f0a700cfcd0 2024-03-18T14:42:42.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.359+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a7004f620 con 0x7f0a700cfcd0 2024-03-18T14:42:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.359+0000 7f0a6e7fc640 1 -- 172.21.15.6:0/1217211810 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7f0a60010040 con 0x7f0a700cfcd0 2024-03-18T14:42:42.362 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:42.362 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:42:42.362 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:42:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.361+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a5c06bbd0 msgr2=0x7f0a5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.361+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a5c06bbd0 0x7f0a5c06e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0a700b0cb0 tx=0x7f0a64009190 comp rx=0 tx=0).stop 2024-03-18T14:42:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.361+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700cfcd0 msgr2=0x7f0a700af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.361+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700cfcd0 0x7f0a700af1e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000ebc0 tx=0x7f0a6000c490 comp rx=0 tx=0).stop 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 shutdown_connections 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a5c06bbd0 0x7f0a5c06e090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a7015bd40 0x7f0a70157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a700d1d90 0x7f0a7015d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 --2- 172.21.15.6:0/1217211810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a700cfcd0 0x7f0a700af1e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 >> 172.21.15.6:0/1217211810 conn(0x7f0a7009ae20 msgr2=0x7f0a700cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 shutdown_connections 2024-03-18T14:42:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.362+0000 7f0a77b97640 1 -- 172.21.15.6:0/1217211810 wait complete. 2024-03-18T14:42:42.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:42.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.474+0000 7f0f142dc640 1 Processor -- start 2024-03-18T14:42:42.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.474+0000 7f0f142dc640 1 -- start start 2024-03-18T14:42:42.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f142dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c15e240 0x7f0f0c06b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f142dc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0f0c0d14d0 0x7f0f0c06b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f142dc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f0c0cce30 con 0x7f0f0c0afd00 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f142dc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0f0c0cb000 con 0x7f0f0c0d14d0 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f142dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0c0b1f90 con 0x7f0f0c15e240 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f11877640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0f0c0afd00 0x7f0f0c0b0170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50110/0 (socket says 172.21.15.6:50110) 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f11877640 1 -- 172.21.15.6:0/105308049 learned_addr learned my addr 172.21.15.6:0/105308049 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f12879640 1 --2- 172.21.15.6:0/105308049 >> v2:172.21.15.92:3300/0 conn(0x7f0f0c0d14d0 0x7f0f0c06b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.475+0000 7f0f12078640 1 --2- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c15e240 0x7f0f0c06b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.476+0000 7f0f11076640 1 -- 172.21.15.6:0/105308049 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 239123991 0 0) 0x7f0f0c0cce30 con 0x7f0f0c0afd00 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.476+0000 7f0f11076640 1 -- 172.21.15.6:0/105308049 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef4003520 con 0x7f0f0c0afd00 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.476+0000 7f0f12879640 1 -- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c15e240 msgr2=0x7f0f0c06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.476+0000 7f0f12879640 1 --2- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c15e240 0x7f0f0c06b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.476+0000 7f0f12879640 1 -- 172.21.15.6:0/105308049 >> v1:172.21.15.6:6789/0 conn(0x7f0f0c0afd00 legacy=0x7f0f0c0b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.476+0000 7f0f12879640 1 -- 172.21.15.6:0/105308049 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c0b1c20 con 0x7f0f0c0d14d0 2024-03-18T14:42:42.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.476+0000 7f0f12078640 1 --2- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c15e240 0x7f0f0c06b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f12879640 1 --2- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c0d14d0 0x7f0f0c06b700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0f040097a0 tx=0x7f0f0402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d76fa9a46a18dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f11076640 1 -- 172.21.15.6:0/105308049 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f040315a0 con 0x7f0f0c0d14d0 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f11076640 1 -- 172.21.15.6:0/105308049 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f04037070 con 0x7f0f0c0d14d0 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f11076640 1 -- 172.21.15.6:0/105308049 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04031960 con 0x7f0f0c0d14d0 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 -- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c0d14d0 msgr2=0x7f0f0c06b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 --2- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c0d14d0 0x7f0f0c06b700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0f040097a0 tx=0x7f0f0402b840 comp rx=0 tx=0).stop 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 -- 172.21.15.6:0/105308049 shutdown_connections 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 --2- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c0d14d0 0x7f0f0c06b700 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 --2- 172.21.15.6:0/105308049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c15e240 0x7f0f0c06b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 -- 172.21.15.6:0/105308049 >> 172.21.15.6:0/105308049 conn(0x7f0f0c0d09b0 msgr2=0x7f0f0c0d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 -- 172.21.15.6:0/105308049 shutdown_connections 2024-03-18T14:42:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.477+0000 7f0f142dc640 1 -- 172.21.15.6:0/105308049 wait complete. 2024-03-18T14:42:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.478+0000 7f0f142dc640 1 Processor -- start 2024-03-18T14:42:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.478+0000 7f0f142dc640 1 -- start start 2024-03-18T14:42:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f142dc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0c0afd00 0x7f0f0c150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f142dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c0d14d0 0x7f0f0c0c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f142dc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c15e240 0x7f0f0c0c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f142dc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c0ce580 con 0x7f0f0c0afd00 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f142dc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c0ce0e0 con 0x7f0f0c15e240 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f142dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0c0ce700 con 0x7f0f0c0d14d0 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f12078640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0c0afd00 0x7f0f0c150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f12078640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0c0afd00 0x7f0f0c150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37152/0 (socket says 172.21.15.6:37152) 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f12078640 1 -- 172.21.15.6:0/3256334116 learned_addr learned my addr 172.21.15.6:0/3256334116 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f12879640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c15e240 0x7f0f0c0c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.479+0000 7f0f11877640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c0d14d0 0x7f0f0c0c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f12078640 1 -- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c0d14d0 msgr2=0x7f0f0c0c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f12078640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c0d14d0 0x7f0f0c0c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f12078640 1 -- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c15e240 msgr2=0x7f0f0c0c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f12078640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c15e240 0x7f0f0c0c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f12078640 1 -- 172.21.15.6:0/3256334116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c0c62b0 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f11877640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c0d14d0 0x7f0f0c0c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f12078640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0c0afd00 0x7f0f0c150560 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00efc0 tx=0x7f0efc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f0affd640 1 -- 172.21.15.6:0/3256334116 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc018020 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f0affd640 1 -- 172.21.15.6:0/3256334116 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0efc014680 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c0c6590 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f0affd640 1 -- 172.21.15.6:0/3256334116 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc014a50 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.480+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c0c6b50 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.481+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0c15a8e0 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.481+0000 7f0f0affd640 1 -- 172.21.15.6:0/3256334116 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0efc005d10 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.482+0000 7f0f0affd640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0ef806bbd0 0x7f0ef806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.482+0000 7f0f0affd640 1 -- 172.21.15.6:0/3256334116 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0efc08bbc0 con 0x7f0f0c0afd00 2024-03-18T14:42:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.482+0000 7f0f11877640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0ef806bbd0 0x7f0ef806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.482+0000 7f0f11877640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0ef806bbd0 0x7f0ef806e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007fd0 tx=0x7f0f00009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.486+0000 7f0f0affd640 1 -- 172.21.15.6:0/3256334116 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0f0c15a8e0 con 0x7f0f0c0afd00 2024-03-18T14:42:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.671+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f0c09e9a0 con 0x7f0f0c0afd00 2024-03-18T14:42:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.673+0000 7f0f0affd640 1 -- 172.21.15.6:0/3256334116 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7f0efc022030 con 0x7f0f0c0afd00 2024-03-18T14:42:42.675 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:42.676 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0ef806bbd0 msgr2=0x7f0ef806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0ef806bbd0 0x7f0ef806e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007fd0 tx=0x7f0f00009040 comp rx=0 tx=0).stop 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0c0afd00 msgr2=0x7f0f0c150560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0c0afd00 0x7f0f0c150560 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00efc0 tx=0x7f0efc00c490 comp rx=0 tx=0).stop 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 shutdown_connections 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0ef806bbd0 0x7f0ef806e090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f0c15e240 0x7f0f0c0c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f0c0d14d0 0x7f0f0c0c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 --2- 172.21.15.6:0/3256334116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f0c0afd00 0x7f0f0c150560 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 >> 172.21.15.6:0/3256334116 conn(0x7f0f0c0d09b0 msgr2=0x7f0f0c06f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 shutdown_connections 2024-03-18T14:42:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.675+0000 7f0f142dc640 1 -- 172.21.15.6:0/3256334116 wait complete. 2024-03-18T14:42:42.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:41:33.458302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:41:33.848049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T14:41:37.567980+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T14:42:42.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:42.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.791+0000 7f52a71cd640 1 Processor -- start 2024-03-18T14:42:42.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.791+0000 7f52a71cd640 1 -- start start 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f52a71cd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52a00d13a0 0x7f52a0099c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f52a71cd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a009a180 0x7f52a00cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f52a71cd640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a00d0300 con 0x7f52a015e150 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f52a71cd640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f52a015e530 con 0x7f52a00d13a0 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f52a71cd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f52a009a5b0 con 0x7f52a009a180 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f529ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52a00d13a0 0x7f52a0099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f529ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52a00d13a0 0x7f52a0099c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37168/0 (socket says 172.21.15.6:37168) 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f529ffff640 1 -- 172.21.15.6:0/693913262 learned_addr learned my addr 172.21.15.6:0/693913262 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f52a576a640 1 --2- 172.21.15.6:0/693913262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a009a180 0x7f52a00cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f529ffff640 1 -- 172.21.15.6:0/693913262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a009a180 msgr2=0x7f52a00cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.792+0000 7f529ffff640 1 --2- 172.21.15.6:0/693913262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a009a180 0x7f52a00cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.793+0000 7f529ffff640 1 -- 172.21.15.6:0/693913262 >> v1:172.21.15.92:6789/0 conn(0x7f52a015e150 legacy=0x7f52a006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.793+0000 7f529ffff640 1 -- 172.21.15.6:0/693913262 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a00d05e0 con 0x7f52a00d13a0 2024-03-18T14:42:42.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.793+0000 7f529ffff640 1 --2- 172.21.15.6:0/693913262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a00d13a0 0x7f52a0099c40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5288009870 tx=0x7f528802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f8a5a7d4a7f78d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.793+0000 7f529f7fe640 1 -- 172.21.15.6:0/693913262 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288035020 con 0x7f52a00d13a0 2024-03-18T14:42:42.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.793+0000 7f529f7fe640 1 -- 172.21.15.6:0/693913262 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5288031640 con 0x7f52a00d13a0 2024-03-18T14:42:42.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.793+0000 7f529f7fe640 1 -- 172.21.15.6:0/693913262 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528802d040 con 0x7f52a00d13a0 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 -- 172.21.15.6:0/693913262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a00d13a0 msgr2=0x7f52a0099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 --2- 172.21.15.6:0/693913262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a00d13a0 0x7f52a0099c40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5288009870 tx=0x7f528802b8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 -- 172.21.15.6:0/693913262 shutdown_connections 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 --2- 172.21.15.6:0/693913262 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a009a180 0x7f52a00cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 --2- 172.21.15.6:0/693913262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a00d13a0 0x7f52a0099c40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 -- 172.21.15.6:0/693913262 >> 172.21.15.6:0/693913262 conn(0x7f52a00b3650 msgr2=0x7f52a00b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 -- 172.21.15.6:0/693913262 shutdown_connections 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.794+0000 7f52a71cd640 1 -- 172.21.15.6:0/693913262 wait complete. 2024-03-18T14:42:42.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.795+0000 7f52a71cd640 1 Processor -- start 2024-03-18T14:42:42.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.795+0000 7f52a71cd640 1 -- start start 2024-03-18T14:42:42.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a71cd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a009a180 0x7f52a0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a71cd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a00d13a0 0x7f52a01f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a71cd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52a015e150 0x7f52a01f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a71cd640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f52a00d0e80 con 0x7f52a009a180 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a71cd640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f52a00d0ba0 con 0x7f52a015e150 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a71cd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f52a006c650 con 0x7f52a00d13a0 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a009a180 0x7f52a0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a009a180 0x7f52a0157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37176/0 (socket says 172.21.15.6:37176) 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 -- 172.21.15.6:0/2380418213 learned_addr learned my addr 172.21.15.6:0/2380418213 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f529ffff640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a00d13a0 0x7f52a01f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a576a640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52a015e150 0x7f52a01f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 -- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a00d13a0 msgr2=0x7f52a01f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a00d13a0 0x7f52a01f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 -- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52a015e150 msgr2=0x7f52a01f79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52a015e150 0x7f52a01f79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.796+0000 7f52a4f69640 1 -- 172.21.15.6:0/2380418213 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a015f0c0 con 0x7f52a009a180 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.797+0000 7f52a4f69640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a009a180 0x7f52a0157ca0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f529800c790 tx=0x7f529800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.797+0000 7f529d7fa640 1 -- 172.21.15.6:0/2380418213 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298016020 con 0x7f52a009a180 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.797+0000 7f529d7fa640 1 -- 172.21.15.6:0/2380418213 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5298006d50 con 0x7f52a009a180 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.797+0000 7f529d7fa640 1 -- 172.21.15.6:0/2380418213 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298003bb0 con 0x7f52a009a180 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.797+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a015f3a0 con 0x7f52a009a180 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.797+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a015f960 con 0x7f52a009a180 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.798+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a00751b0 con 0x7f52a009a180 2024-03-18T14:42:42.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.798+0000 7f529d7fa640 1 -- 172.21.15.6:0/2380418213 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f529800a850 con 0x7f52a009a180 2024-03-18T14:42:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.799+0000 7f529d7fa640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f528006bc00 0x7f528006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.799+0000 7f529d7fa640 1 -- 172.21.15.6:0/2380418213 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f529808aa50 con 0x7f52a009a180 2024-03-18T14:42:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.799+0000 7f529ffff640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f528006bc00 0x7f528006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:42.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.799+0000 7f529ffff640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f528006bc00 0x7f528006e0c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5288036000 tx=0x7f528803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.803+0000 7f529d7fa640 1 -- 172.21.15.6:0/2380418213 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5298020020 con 0x7f52a009a180 2024-03-18T14:42:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.983+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f52a00cf7d0 con 0x7f52a009a180 2024-03-18T14:42:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.983+0000 7f529d7fa640 1 -- 172.21.15.6:0/2380418213 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f5298058ae0 con 0x7f52a009a180 2024-03-18T14:42:42.986 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f528006bc00 msgr2=0x7f528006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f528006bc00 0x7f528006e0c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5288036000 tx=0x7f528803c040 comp rx=0 tx=0).stop 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a009a180 msgr2=0x7f52a0157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a009a180 0x7f52a0157ca0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f529800c790 tx=0x7f529800cc50 comp rx=0 tx=0).stop 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 shutdown_connections 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f528006bc00 0x7f528006e0c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52a015e150 0x7f52a01f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52a00d13a0 0x7f52a01f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 --2- 172.21.15.6:0/2380418213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52a009a180 0x7f52a0157ca0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 >> 172.21.15.6:0/2380418213 conn(0x7f52a00b3650 msgr2=0x7f52a00cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 shutdown_connections 2024-03-18T14:42:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:42.985+0000 7f52a71cd640 1 -- 172.21.15.6:0/2380418213 wait complete. 2024-03-18T14:42:42.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:42:43.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.100+0000 7f539573b640 1 Processor -- start 2024-03-18T14:42:43.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.100+0000 7f539573b640 1 -- start start 2024-03-18T14:42:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f539573b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f539009ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f539573b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f539009a3c0 0x7f53900cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f539573b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53900b4920 con 0x7f539006cef0 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f539573b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f53900cc120 con 0x7f539009a3c0 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f539573b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f539006d740 con 0x7f5390161740 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f538effd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f539009ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f538f7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f539009a3c0 0x7f53900cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f538e7fc640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f539006cef0 0x7f5390099cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48594/0 (socket says 172.21.15.6:48594) 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.101+0000 7f538e7fc640 1 -- 172.21.15.6:0/3767941012 learned_addr learned my addr 172.21.15.6:0/3767941012 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.102+0000 7f538f7fe640 1 -- 172.21.15.6:0/3767941012 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 msgr2=0x7f539009ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.102+0000 7f538f7fe640 1 --2- 172.21.15.6:0/3767941012 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f539009ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.102+0000 7f538f7fe640 1 -- 172.21.15.6:0/3767941012 >> v1:172.21.15.92:6789/0 conn(0x7f539006cef0 legacy=0x7f5390099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.102+0000 7f538f7fe640 1 -- 172.21.15.6:0/3767941012 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53900cc370 con 0x7f539009a3c0 2024-03-18T14:42:43.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.102+0000 7f538dffb640 1 -- 172.21.15.6:0/3767941012 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 604844815 0 0) 0x7f5384001110 con 0x7f539006cef0 2024-03-18T14:42:43.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f538f7fe640 1 --2- 172.21.15.6:0/3767941012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539009a3c0 0x7f53900cce50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5380009d90 tx=0x7f538002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ef74adffecca62a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f538dffb640 1 -- 172.21.15.6:0/3767941012 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380039070 con 0x7f539009a3c0 2024-03-18T14:42:43.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f538dffb640 1 -- 172.21.15.6:0/3767941012 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f538002cc50 con 0x7f539009a3c0 2024-03-18T14:42:43.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f538dffb640 1 -- 172.21.15.6:0/3767941012 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380034440 con 0x7f539009a3c0 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 -- 172.21.15.6:0/3767941012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539009a3c0 msgr2=0x7f53900cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 --2- 172.21.15.6:0/3767941012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539009a3c0 0x7f53900cce50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5380009d90 tx=0x7f538002bea0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 -- 172.21.15.6:0/3767941012 shutdown_connections 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 --2- 172.21.15.6:0/3767941012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539009a3c0 0x7f53900cce50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 --2- 172.21.15.6:0/3767941012 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f539009ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 -- 172.21.15.6:0/3767941012 >> 172.21.15.6:0/3767941012 conn(0x7f53900b35c0 msgr2=0x7f53900b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 -- 172.21.15.6:0/3767941012 shutdown_connections 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.103+0000 7f539573b640 1 -- 172.21.15.6:0/3767941012 wait complete. 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 Processor -- start 2024-03-18T14:42:43.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 -- start start 2024-03-18T14:42:43.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539006cef0 0x7f53900cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f539009a3c0 0x7f53900c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f5390160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f539006d330 con 0x7f539009a3c0 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53900cc930 con 0x7f539006cef0 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f539573b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f539006d4b0 con 0x7f5390161740 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f538e7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f539009a3c0 0x7f53900c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f538f7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f5390160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f538e7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f539009a3c0 0x7f53900c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37178/0 (socket says 172.21.15.6:37178) 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f538e7fc640 1 -- 172.21.15.6:0/2876118373 learned_addr learned my addr 172.21.15.6:0/2876118373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.105+0000 7f538effd640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539006cef0 0x7f53900cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538f7fe640 1 -- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539006cef0 msgr2=0x7f53900cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538f7fe640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539006cef0 0x7f53900cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538f7fe640 1 -- 172.21.15.6:0/2876118373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f539009a3c0 msgr2=0x7f53900c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538f7fe640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f539009a3c0 0x7f53900c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538f7fe640 1 -- 172.21.15.6:0/2876118373 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53900c9940 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538e7fc640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f539009a3c0 0x7f53900c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538effd640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539006cef0 0x7f53900cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f538f7fe640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f5390160610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5380033000 tx=0x7f5380004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f53777fe640 1 -- 172.21.15.6:0/2876118373 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380039050 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f53777fe640 1 -- 172.21.15.6:0/2876118373 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5380045070 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f53777fe640 1 -- 172.21.15.6:0/2876118373 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380009450 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53900c9660 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.106+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f53901f72e0 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.107+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53900751b0 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.107+0000 7f53777fe640 1 -- 172.21.15.6:0/2876118373 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f538003e610 con 0x7f5390161740 2024-03-18T14:42:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.109+0000 7f53777fe640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f537c06bbd0 0x7f537c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.109+0000 7f53777fe640 1 -- 172.21.15.6:0/2876118373 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f53800ad720 con 0x7f5390161740 2024-03-18T14:42:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.109+0000 7f538effd640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f537c06bbd0 0x7f537c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.109+0000 7f538effd640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f537c06bbd0 0x7f537c06e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5388009e90 tx=0x7f5388009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.113+0000 7f53777fe640 1 -- 172.21.15.6:0/2876118373 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f538007b860 con 0x7f5390161740 2024-03-18T14:42:43.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.300+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f539006eca0 con 0x7f5390161740 2024-03-18T14:42:43.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.300+0000 7f53777fe640 1 -- 172.21.15.6:0/2876118373 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f538007f900 con 0x7f5390161740 2024-03-18T14:42:43.305 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f537c06bbd0 msgr2=0x7f537c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f537c06bbd0 0x7f537c06e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5388009e90 tx=0x7f5388009190 comp rx=0 tx=0).stop 2024-03-18T14:42:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 msgr2=0x7f5390160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f5390160610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5380033000 tx=0x7f5380004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 shutdown_connections 2024-03-18T14:42:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f537c06bbd0 0x7f537c06e090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5390161740 0x7f5390160610 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f539009a3c0 0x7f53900c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 --2- 172.21.15.6:0/2876118373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f539006cef0 0x7f53900cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 >> 172.21.15.6:0/2876118373 conn(0x7f53900b35c0 msgr2=0x7f53900d0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 shutdown_connections 2024-03-18T14:42:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.302+0000 7f539573b640 1 -- 172.21.15.6:0/2876118373 wait complete. 2024-03-18T14:42:43.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:42:43.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.417+0000 7fe10ef9d640 1 Processor -- start 2024-03-18T14:42:43.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.417+0000 7fe10ef9d640 1 -- start start 2024-03-18T14:42:43.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10ef9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe1080cbd40 0x7fe1080b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10ef9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe10806f5e0 0x7fe10806fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10ef9d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1080cc330 con 0x7fe1080ce820 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10ef9d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe1080cc610 con 0x7fe1080cbd40 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10ef9d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe10806eb00 con 0x7fe10806f5e0 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10df9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe1080ce820 0x7fe1080cec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50120/0 (socket says 172.21.15.6:50120) 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10df9b640 1 -- 172.21.15.6:0/2613718419 learned_addr learned my addr 172.21.15.6:0/2613718419 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10d79a640 1 --2- 172.21.15.6:0/2613718419 >> v2:172.21.15.92:3300/0 conn(0x7fe1080cbd40 0x7fe1080b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.418+0000 7fe10e79c640 1 --2- 172.21.15.6:0/2613718419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe10806f5e0 0x7fe10806fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.419+0000 7fe10cf99640 1 -- 172.21.15.6:0/2613718419 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4287033068 0 0) 0x7fe1080cc330 con 0x7fe1080ce820 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.419+0000 7fe10cf99640 1 -- 172.21.15.6:0/2613718419 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0f0003520 con 0x7fe1080ce820 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.419+0000 7fe10e79c640 1 -- 172.21.15.6:0/2613718419 >> v2:172.21.15.92:3300/0 conn(0x7fe1080cbd40 msgr2=0x7fe1080b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.419+0000 7fe10e79c640 1 --2- 172.21.15.6:0/2613718419 >> v2:172.21.15.92:3300/0 conn(0x7fe1080cbd40 0x7fe1080b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.419+0000 7fe10e79c640 1 -- 172.21.15.6:0/2613718419 >> v1:172.21.15.6:6789/0 conn(0x7fe1080ce820 legacy=0x7fe1080cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.419+0000 7fe10e79c640 1 -- 172.21.15.6:0/2613718419 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10806ede0 con 0x7fe10806f5e0 2024-03-18T14:42:43.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.419+0000 7fe10d79a640 1 --2- 172.21.15.6:0/2613718419 >> v2:172.21.15.92:3300/0 conn(0x7fe1080cbd40 0x7fe1080b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10e79c640 1 --2- 172.21.15.6:0/2613718419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe10806f5e0 0x7fe10806fa70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009d90 tx=0x7fe0f802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf3297ab6866c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10cf99640 1 -- 172.21.15.6:0/2613718419 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8039070 con 0x7fe10806f5e0 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10cf99640 1 -- 172.21.15.6:0/2613718419 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f802cc50 con 0x7fe10806f5e0 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10cf99640 1 -- 172.21.15.6:0/2613718419 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8034440 con 0x7fe10806f5e0 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 -- 172.21.15.6:0/2613718419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe10806f5e0 msgr2=0x7fe10806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/2613718419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe10806f5e0 0x7fe10806fa70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009d90 tx=0x7fe0f802bea0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 -- 172.21.15.6:0/2613718419 shutdown_connections 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/2613718419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe10806f5e0 0x7fe10806fa70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/2613718419 >> v2:172.21.15.92:3300/0 conn(0x7fe1080cbd40 0x7fe1080b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 -- 172.21.15.6:0/2613718419 >> 172.21.15.6:0/2613718419 conn(0x7fe1080b2310 msgr2=0x7fe1080b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 -- 172.21.15.6:0/2613718419 shutdown_connections 2024-03-18T14:42:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.420+0000 7fe10ef9d640 1 -- 172.21.15.6:0/2613718419 wait complete. 2024-03-18T14:42:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 Processor -- start 2024-03-18T14:42:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 -- start start 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe10806f5e0 0x7fe10814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1080cbd40 0x7fe108150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1080ce820 0x7fe1080ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe1080cf630 con 0x7fe1080cbd40 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe10815e210 con 0x7fe10806f5e0 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10ef9d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe10814f530 con 0x7fe1080ce820 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10d79a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1080cbd40 0x7fe108150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10d79a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1080cbd40 0x7fe108150820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37182/0 (socket says 172.21.15.6:37182) 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10d79a640 1 -- 172.21.15.6:0/3300925780 learned_addr learned my addr 172.21.15.6:0/3300925780 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10df9b640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe10806f5e0 0x7fe10814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.422+0000 7fe10e79c640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1080ce820 0x7fe1080ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10d79a640 1 -- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1080ce820 msgr2=0x7fe1080ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10d79a640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1080ce820 0x7fe1080ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10d79a640 1 -- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe10806f5e0 msgr2=0x7fe10814dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10d79a640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe10806f5e0 0x7fe10814dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10d79a640 1 -- 172.21.15.6:0/3300925780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10814f780 con 0x7fe1080cbd40 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10d79a640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1080cbd40 0x7fe108150820 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00ebe0 tx=0x7fe0fc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe1067fc640 1 -- 172.21.15.6:0/3300925780 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc017070 con 0x7fe1080cbd40 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe1067fc640 1 -- 172.21.15.6:0/3300925780 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0fc00cc30 con 0x7fe1080cbd40 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10814fa60 con 0x7fe1080cbd40 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe1067fc640 1 -- 172.21.15.6:0/3300925780 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc013470 con 0x7fe1080cbd40 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.423+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe108150020 con 0x7fe1080cbd40 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.424+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe10806d2d0 con 0x7fe1080cbd40 2024-03-18T14:42:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.424+0000 7fe1067fc640 1 -- 172.21.15.6:0/3300925780 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe0fc01a070 con 0x7fe1080cbd40 2024-03-18T14:42:43.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.425+0000 7fe1067fc640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0f406bbd0 0x7fe0f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.425+0000 7fe1067fc640 1 -- 172.21.15.6:0/3300925780 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe0fc08b0b0 con 0x7fe1080cbd40 2024-03-18T14:42:43.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.425+0000 7fe10df9b640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0f406bbd0 0x7fe0f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.425+0000 7fe10df9b640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0f406bbd0 0x7fe0f406e090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe100005880 tx=0x7fe100009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.430+0000 7fe1067fc640 1 -- 172.21.15.6:0/3300925780 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe0fc059150 con 0x7fe1080cbd40 2024-03-18T14:42:43.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.615+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe10804f540 con 0x7fe1080cbd40 2024-03-18T14:42:43.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.616+0000 7fe1067fc640 1 -- 172.21.15.6:0/3300925780 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fe0fc05d1f0 con 0x7fe1080cbd40 2024-03-18T14:42:43.618 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0f406bbd0 msgr2=0x7fe0f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0f406bbd0 0x7fe0f406e090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe100005880 tx=0x7fe100009040 comp rx=0 tx=0).stop 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1080cbd40 msgr2=0x7fe108150820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1080cbd40 0x7fe108150820 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00ebe0 tx=0x7fe0fc00c520 comp rx=0 tx=0).stop 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 shutdown_connections 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0f406bbd0 0x7fe0f406e090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1080ce820 0x7fe1080ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1080cbd40 0x7fe108150820 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 --2- 172.21.15.6:0/3300925780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe10806f5e0 0x7fe10814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 >> 172.21.15.6:0/3300925780 conn(0x7fe1080b2310 msgr2=0x7fe10806bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 shutdown_connections 2024-03-18T14:42:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.618+0000 7fe10ef9d640 1 -- 172.21.15.6:0/3300925780 wait complete. 2024-03-18T14:42:43.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T14:42:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.733+0000 7f601e08c640 1 Processor -- start 2024-03-18T14:42:43.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.733+0000 7f601e08c640 1 -- start start 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f601e08c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60180d2e50 0x7f60180b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f601e08c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60180b5c80 0x7f60180b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f601e08c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601815e3b0 con 0x7f601815dfd0 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f601e08c640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f60180b8470 con 0x7f60180b5c80 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f601e08c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f60180b8750 con 0x7f60180d2e50 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6017fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60180b5c80 0x7f60180b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6017fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60180b5c80 0x7f60180b6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50752/0 (socket says 172.21.15.6:50752) 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6016ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60180d2e50 0x7f60180b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6017fff640 1 -- 172.21.15.6:0/3517490345 learned_addr learned my addr 172.21.15.6:0/3517490345 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:43.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f60177fe640 1 --1- 172.21.15.6:0/3517490345 >> v1:172.21.15.92:6789/0 conn(0x7f601815dfd0 0x7f60180b9650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40472/0 (socket says 172.21.15.6:40472) 2024-03-18T14:42:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6016ffd640 1 -- 172.21.15.6:0/3517490345 >> v2:172.21.15.6:3300/0 conn(0x7f60180b5c80 msgr2=0x7f60180b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6016ffd640 1 --2- 172.21.15.6:0/3517490345 >> v2:172.21.15.6:3300/0 conn(0x7f60180b5c80 0x7f60180b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6016ffd640 1 -- 172.21.15.6:0/3517490345 >> v1:172.21.15.92:6789/0 conn(0x7f601815dfd0 legacy=0x7f60180b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6016ffd640 1 -- 172.21.15.6:0/3517490345 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60180b8a30 con 0x7f60180d2e50 2024-03-18T14:42:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f60167fc640 1 -- 172.21.15.6:0/3517490345 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3454411153 0 0) 0x7f601815e3b0 con 0x7f601815dfd0 2024-03-18T14:42:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f6016ffd640 1 --2- 172.21.15.6:0/3517490345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180d2e50 0x7f60180b5740 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60180ce4a0 tx=0x7f600c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e1eefc4e9a474ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f60167fc640 1 -- 172.21.15.6:0/3517490345 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c035020 con 0x7f60180d2e50 2024-03-18T14:42:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f60167fc640 1 -- 172.21.15.6:0/3517490345 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f600c031830 con 0x7f60180d2e50 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.735+0000 7f60167fc640 1 -- 172.21.15.6:0/3517490345 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c031c00 con 0x7f60180d2e50 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 -- 172.21.15.6:0/3517490345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180d2e50 msgr2=0x7f60180b5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 --2- 172.21.15.6:0/3517490345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180d2e50 0x7f60180b5740 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60180ce4a0 tx=0x7f600c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 -- 172.21.15.6:0/3517490345 shutdown_connections 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 --2- 172.21.15.6:0/3517490345 >> v2:172.21.15.6:3300/0 conn(0x7f60180b5c80 0x7f60180b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 --2- 172.21.15.6:0/3517490345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180d2e50 0x7f60180b5740 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 -- 172.21.15.6:0/3517490345 >> 172.21.15.6:0/3517490345 conn(0x7f6018072d10 msgr2=0x7f6018073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 -- 172.21.15.6:0/3517490345 shutdown_connections 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.736+0000 7f601e08c640 1 -- 172.21.15.6:0/3517490345 wait complete. 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.737+0000 7f601e08c640 1 Processor -- start 2024-03-18T14:42:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.737+0000 7f601e08c640 1 -- start start 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f601e08c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180b5c80 0x7f6018159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f601e08c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60180d2e50 0x7f60181f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f601e08c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601815dfd0 0x7f60181f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f601e08c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60180b74b0 con 0x7f601815dfd0 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f601e08c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f60180b7330 con 0x7f60180b5c80 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f601e08c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f60180b7630 con 0x7f60180d2e50 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601815dfd0 0x7f60181f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601815dfd0 0x7f60181f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50766/0 (socket says 172.21.15.6:50766) 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 -- 172.21.15.6:0/825449164 learned_addr learned my addr 172.21.15.6:0/825449164 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f60177fe640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180b5c80 0x7f6018159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6016ffd640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60180d2e50 0x7f60181f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 -- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60180d2e50 msgr2=0x7f60181f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60180d2e50 0x7f60181f3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 -- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180b5c80 msgr2=0x7f6018159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180b5c80 0x7f6018159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 -- 172.21.15.6:0/825449164 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60181f4530 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f60177fe640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180b5c80 0x7f6018159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.738+0000 7f6017fff640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601815dfd0 0x7f60181f3bb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6010009d90 tx=0x7f601000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.739+0000 7f5ffffff640 1 -- 172.21.15.6:0/825449164 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601000cd20 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.739+0000 7f5ffffff640 1 -- 172.21.15.6:0/825449164 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6010003990 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.739+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60181f4250 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.739+0000 7f5ffffff640 1 -- 172.21.15.6:0/825449164 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010003d60 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.739+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60181f4b20 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.740+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60180751b0 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.740+0000 7f5ffffff640 1 -- 172.21.15.6:0/825449164 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6010005d10 con 0x7f601815dfd0 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.740+0000 7f5ffffff640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f600406bbd0 0x7f600406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.742+0000 7f5ffffff640 1 -- 172.21.15.6:0/825449164 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f601008b2d0 con 0x7f601815dfd0 2024-03-18T14:42:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.742+0000 7f60177fe640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f600406bbd0 0x7f600406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.742+0000 7f60177fe640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f600406bbd0 0x7f600406e090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6008007fd0 tx=0x7f6008007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.745+0000 7f5ffffff640 1 -- 172.21.15.6:0/825449164 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6010022020 con 0x7f601815dfd0 2024-03-18T14:42:43.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.928+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6018159ba0 con 0x7f601815dfd0 2024-03-18T14:42:43.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.928+0000 7f5ffffff640 1 -- 172.21.15.6:0/825449164 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f6010010040 con 0x7f601815dfd0 2024-03-18T14:42:43.931 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f600406bbd0 msgr2=0x7f600406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f600406bbd0 0x7f600406e090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6008007fd0 tx=0x7f6008007de0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601815dfd0 msgr2=0x7f60181f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601815dfd0 0x7f60181f3bb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6010009d90 tx=0x7f601000c490 comp rx=0 tx=0).stop 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 shutdown_connections 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f600406bbd0 0x7f600406e090 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601815dfd0 0x7f60181f3bb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60180d2e50 0x7f60181f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 --2- 172.21.15.6:0/825449164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60180b5c80 0x7f6018159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:43.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 >> 172.21.15.6:0/825449164 conn(0x7f6018072d10 msgr2=0x7f60180cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:43.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 shutdown_connections 2024-03-18T14:42:43.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:43.930+0000 7f601e08c640 1 -- 172.21.15.6:0/825449164 wait complete. 2024-03-18T14:42:43.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:44.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.042+0000 7fd04a597640 1 Processor -- start 2024-03-18T14:42:44.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.042+0000 7fd04a597640 1 -- start start 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd04a597640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0440b18e0 0x7fd0440b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd04a597640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd04406b1c0 0x7fd04415c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd04a597640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0440b2200 con 0x7fd04415f850 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd04a597640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd0440cb220 con 0x7fd04406b1c0 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd04a597640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd0440cc9e0 con 0x7fd0440b18e0 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd043fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0440b18e0 0x7fd0440b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd0437fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd04406b1c0 0x7fd04415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd043fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0440b18e0 0x7fd0440b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55056/0 (socket says 172.21.15.6:55056) 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd043fff640 1 -- 172.21.15.6:0/50149461 learned_addr learned my addr 172.21.15.6:0/50149461 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd048b34640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd04415f850 0x7fd04415fce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40484/0 (socket says 172.21.15.6:40484) 2024-03-18T14:42:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd043fff640 1 -- 172.21.15.6:0/50149461 >> v2:172.21.15.92:3300/0 conn(0x7fd04406b1c0 msgr2=0x7fd04415c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:44.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd043fff640 1 --2- 172.21.15.6:0/50149461 >> v2:172.21.15.92:3300/0 conn(0x7fd04406b1c0 0x7fd04415c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd043fff640 1 -- 172.21.15.6:0/50149461 >> v1:172.21.15.92:6789/0 conn(0x7fd04415f850 legacy=0x7fd04415fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.043+0000 7fd043fff640 1 -- 172.21.15.6:0/50149461 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04415cab0 con 0x7fd0440b18e0 2024-03-18T14:42:44.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.044+0000 7fd043fff640 1 --2- 172.21.15.6:0/50149461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0440b18e0 0x7fd0440b1cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd038009870 tx=0x7fd03802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8897dac5310b5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.044+0000 7fd042ffd640 1 -- 172.21.15.6:0/50149461 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038034070 con 0x7fd0440b18e0 2024-03-18T14:42:44.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.044+0000 7fd042ffd640 1 -- 172.21.15.6:0/50149461 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038030840 con 0x7fd0440b18e0 2024-03-18T14:42:44.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.044+0000 7fd042ffd640 1 -- 172.21.15.6:0/50149461 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038030c10 con 0x7fd0440b18e0 2024-03-18T14:42:44.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 -- 172.21.15.6:0/50149461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0440b18e0 msgr2=0x7fd0440b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 --2- 172.21.15.6:0/50149461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0440b18e0 0x7fd0440b1cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd038009870 tx=0x7fd03802b980 comp rx=0 tx=0).stop 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 -- 172.21.15.6:0/50149461 shutdown_connections 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 --2- 172.21.15.6:0/50149461 >> v2:172.21.15.92:3300/0 conn(0x7fd04406b1c0 0x7fd04415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 --2- 172.21.15.6:0/50149461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0440b18e0 0x7fd0440b1cc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 -- 172.21.15.6:0/50149461 >> 172.21.15.6:0/50149461 conn(0x7fd0440d09a0 msgr2=0x7fd0440d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 -- 172.21.15.6:0/50149461 shutdown_connections 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.045+0000 7fd04a597640 1 -- 172.21.15.6:0/50149461 wait complete. 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.046+0000 7fd04a597640 1 Processor -- start 2024-03-18T14:42:44.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.046+0000 7fd04a597640 1 -- start start 2024-03-18T14:42:44.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd04406b1c0 0x7fd0440c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0440b18e0 0x7fd044150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd04415f850 0x7fd04406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0440cc640 con 0x7fd04415f850 2024-03-18T14:42:44.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd04415d070 con 0x7fd0440b18e0 2024-03-18T14:42:44.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd04415c7a0 con 0x7fd04406b1c0 2024-03-18T14:42:44.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd04406b1c0 0x7fd0440c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd048b34640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd04415f850 0x7fd04406f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd04406b1c0 0x7fd0440c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55072/0 (socket says 172.21.15.6:55072) 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 -- 172.21.15.6:0/1002889784 learned_addr learned my addr 172.21.15.6:0/1002889784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd0437fe640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0440b18e0 0x7fd044150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 -- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0440b18e0 msgr2=0x7fd044150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0440b18e0 0x7fd044150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 -- 172.21.15.6:0/1002889784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd04415f850 msgr2=0x7fd04406f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd04415f850 0x7fd04406f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 -- 172.21.15.6:0/1002889784 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0440ce570 con 0x7fd04406b1c0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd043fff640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd04406b1c0 0x7fd0440c8ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd038004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd040ff9640 1 -- 172.21.15.6:0/1002889784 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038048070 con 0x7fd04406b1c0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd040ff9640 1 -- 172.21.15.6:0/1002889784 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038030640 con 0x7fd04406b1c0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd040ff9640 1 -- 172.21.15.6:0/1002889784 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038034040 con 0x7fd04406b1c0 2024-03-18T14:42:44.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0440c90a0 con 0x7fd04406b1c0 2024-03-18T14:42:44.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.048+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0440c9660 con 0x7fd04406b1c0 2024-03-18T14:42:44.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.049+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0440b46b0 con 0x7fd04406b1c0 2024-03-18T14:42:44.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.049+0000 7fd040ff9640 1 -- 172.21.15.6:0/1002889784 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd03804b070 con 0x7fd04406b1c0 2024-03-18T14:42:44.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.049+0000 7fd040ff9640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd03006bbd0 0x7fd03006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.049+0000 7fd040ff9640 1 -- 172.21.15.6:0/1002889784 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd0380acd10 con 0x7fd04406b1c0 2024-03-18T14:42:44.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.049+0000 7fd0437fe640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd03006bbd0 0x7fd03006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.050+0000 7fd0437fe640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd03006bbd0 0x7fd03006e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd0440ca3c0 tx=0x7fd034009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.054+0000 7fd040ff9640 1 -- 172.21.15.6:0/1002889784 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd0380b2020 con 0x7fd04406b1c0 2024-03-18T14:42:44.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.275+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd04409ea10 con 0x7fd04406b1c0 2024-03-18T14:42:44.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.275+0000 7fd040ff9640 1 -- 172.21.15.6:0/1002889784 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7fd0440ce570 con 0x7fd04406b1c0 2024-03-18T14:42:44.278 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:44.278 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:42:44.278 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:42:44.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd03006bbd0 msgr2=0x7fd03006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd03006bbd0 0x7fd03006e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd0440ca3c0 tx=0x7fd034009190 comp rx=0 tx=0).stop 2024-03-18T14:42:44.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd04406b1c0 msgr2=0x7fd0440c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd04406b1c0 0x7fd0440c8ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd038004730 comp rx=0 tx=0).stop 2024-03-18T14:42:44.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 shutdown_connections 2024-03-18T14:42:44.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd03006bbd0 0x7fd03006e090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd04415f850 0x7fd04406f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0440b18e0 0x7fd044150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 --2- 172.21.15.6:0/1002889784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd04406b1c0 0x7fd0440c8ac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 >> 172.21.15.6:0/1002889784 conn(0x7fd0440d09a0 msgr2=0x7fd0440703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:44.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 shutdown_connections 2024-03-18T14:42:44.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.277+0000 7fd04a597640 1 -- 172.21.15.6:0/1002889784 wait complete. 2024-03-18T14:42:44.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:44.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.390+0000 7efd7e3b3640 1 Processor -- start 2024-03-18T14:42:44.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.390+0000 7efd7e3b3640 1 -- start start 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd7e3b3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd780ae840 0x7efd780cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd7e3b3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd780cb680 0x7efd780cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd7e3b3640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd780cc000 con 0x7efd7815ab10 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd7e3b3640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efd780b58c0 con 0x7efd780ae840 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd7e3b3640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7efd7806fca0 con 0x7efd780cb680 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd780ae840 0x7efd780cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd780ae840 0x7efd780cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50782/0 (socket says 172.21.15.6:50782) 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 -- 172.21.15.6:0/2992939224 learned_addr learned my addr 172.21.15.6:0/2992939224 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd7c950640 1 --2- 172.21.15.6:0/2992939224 >> v2:172.21.15.92:3300/0 conn(0x7efd780cb680 0x7efd780cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 -- 172.21.15.6:0/2992939224 >> v2:172.21.15.92:3300/0 conn(0x7efd780cb680 msgr2=0x7efd780cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 --2- 172.21.15.6:0/2992939224 >> v2:172.21.15.92:3300/0 conn(0x7efd780cb680 0x7efd780cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 -- 172.21.15.6:0/2992939224 >> v1:172.21.15.92:6789/0 conn(0x7efd7815ab10 legacy=0x7efd780b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 -- 172.21.15.6:0/2992939224 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7806f120 con 0x7efd780ae840 2024-03-18T14:42:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd76ffd640 1 -- 172.21.15.6:0/2992939224 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532165779 0 0) 0x7efd60001110 con 0x7efd7815ab10 2024-03-18T14:42:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd7c950640 1 --2- 172.21.15.6:0/2992939224 >> v2:172.21.15.92:3300/0 conn(0x7efd780cb680 0x7efd780cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd777fe640 1 --2- 172.21.15.6:0/2992939224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd780cd450 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efd780c68d0 tx=0x7efd6802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72701770709ae2b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd76ffd640 1 -- 172.21.15.6:0/2992939224 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68034070 con 0x7efd780ae840 2024-03-18T14:42:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd76ffd640 1 -- 172.21.15.6:0/2992939224 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd68030860 con 0x7efd780ae840 2024-03-18T14:42:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.392+0000 7efd76ffd640 1 -- 172.21.15.6:0/2992939224 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68030c30 con 0x7efd780ae840 2024-03-18T14:42:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 -- 172.21.15.6:0/2992939224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 msgr2=0x7efd780cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/2992939224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd780cd450 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efd780c68d0 tx=0x7efd6802b9a0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 -- 172.21.15.6:0/2992939224 shutdown_connections 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/2992939224 >> v2:172.21.15.92:3300/0 conn(0x7efd780cb680 0x7efd780cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/2992939224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd780cd450 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 -- 172.21.15.6:0/2992939224 >> 172.21.15.6:0/2992939224 conn(0x7efd780b1470 msgr2=0x7efd780b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 -- 172.21.15.6:0/2992939224 shutdown_connections 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.393+0000 7efd7e3b3640 1 -- 172.21.15.6:0/2992939224 wait complete. 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.394+0000 7efd7e3b3640 1 Processor -- start 2024-03-18T14:42:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.394+0000 7efd7e3b3640 1 -- start start 2024-03-18T14:42:44.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd7e3b3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd781f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd7e3b3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd780cb680 0x7efd781533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd7e3b3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7815ab10 0x7efd780acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd7e3b3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efd7806f9c0 con 0x7efd780ae840 2024-03-18T14:42:44.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd7e3b3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efd7806f6e0 con 0x7efd7815ab10 2024-03-18T14:42:44.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd7e3b3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efd7815aef0 con 0x7efd780cb680 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd781f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd781f51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50784/0 (socket says 172.21.15.6:50784) 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 -- 172.21.15.6:0/845800947 learned_addr learned my addr 172.21.15.6:0/845800947 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd7c950640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7815ab10 0x7efd780acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd777fe640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd780cb680 0x7efd781533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 -- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd780cb680 msgr2=0x7efd781533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd780cb680 0x7efd781533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 -- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7815ab10 msgr2=0x7efd780acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7815ab10 0x7efd780acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 -- 172.21.15.6:0/845800947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd781f5700 con 0x7efd780ae840 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd77fff640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd781f51c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c440 tx=0x7efd6000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.395+0000 7efd74ff9640 1 -- 172.21.15.6:0/845800947 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60017070 con 0x7efd780ae840 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.396+0000 7efd74ff9640 1 -- 172.21.15.6:0/845800947 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd600035f0 con 0x7efd780ae840 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.396+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd780c6ca0 con 0x7efd780ae840 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.396+0000 7efd74ff9640 1 -- 172.21.15.6:0/845800947 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600039e0 con 0x7efd780ae840 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.396+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd780c7260 con 0x7efd780ae840 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.397+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd78073310 con 0x7efd780ae840 2024-03-18T14:42:44.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.397+0000 7efd74ff9640 1 -- 172.21.15.6:0/845800947 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efd6001a070 con 0x7efd780ae840 2024-03-18T14:42:44.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.397+0000 7efd74ff9640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd5806bbd0 0x7efd5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.398+0000 7efd74ff9640 1 -- 172.21.15.6:0/845800947 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7efd6008a1f0 con 0x7efd780ae840 2024-03-18T14:42:44.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.398+0000 7efd777fe640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd5806bbd0 0x7efd5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.398+0000 7efd777fe640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd5806bbd0 0x7efd5806e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efd68036000 tx=0x7efd68008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.403+0000 7efd74ff9640 1 -- 172.21.15.6:0/845800947 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efd780cc000 con 0x7efd780ae840 2024-03-18T14:42:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.585+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd7804f620 con 0x7efd780ae840 2024-03-18T14:42:44.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.586+0000 7efd74ff9640 1 -- 172.21.15.6:0/845800947 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7efd60058200 con 0x7efd780ae840 2024-03-18T14:42:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:44.589 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:44.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd5806bbd0 msgr2=0x7efd5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd5806bbd0 0x7efd5806e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efd68036000 tx=0x7efd68008030 comp rx=0 tx=0).stop 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 msgr2=0x7efd781f51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd781f51c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c440 tx=0x7efd6000c840 comp rx=0 tx=0).stop 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 shutdown_connections 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efd5806bbd0 0x7efd5806e090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7815ab10 0x7efd780acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd780cb680 0x7efd781533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 --2- 172.21.15.6:0/845800947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd780ae840 0x7efd781f51c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 >> 172.21.15.6:0/845800947 conn(0x7efd780b1470 msgr2=0x7efd780aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 shutdown_connections 2024-03-18T14:42:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.589+0000 7efd7e3b3640 1 -- 172.21.15.6:0/845800947 wait complete. 2024-03-18T14:42:44.601 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T14:42:44.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:44.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f7f5b640 1 Processor -- start 2024-03-18T14:42:44.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f7f5b640 1 -- start start 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f7f5b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f91f00d17f0 0x7f91f00b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f7f5b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f91f00cc990 0x7f91f00cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f7f5b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f00b1eb0 con 0x7f91f00d05d0 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f7f5b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f91f00b2190 con 0x7f91f00d17f0 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f7f5b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f91f00b2470 con 0x7f91f00cc990 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f5cf7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f91f00d17f0 0x7f91f00b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f5cf7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f91f00d17f0 0x7f91f00b17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50798/0 (socket says 172.21.15.6:50798) 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f5cf7640 1 -- 172.21.15.6:0/3665570232 learned_addr learned my addr 172.21.15.6:0/3665570232 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f64f8640 1 --2- 172.21.15.6:0/3665570232 >> v2:172.21.15.92:3300/0 conn(0x7f91f00cc990 0x7f91f00cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f5cf7640 1 -- 172.21.15.6:0/3665570232 >> v2:172.21.15.92:3300/0 conn(0x7f91f00cc990 msgr2=0x7f91f00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:44.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f5cf7640 1 --2- 172.21.15.6:0/3665570232 >> v2:172.21.15.92:3300/0 conn(0x7f91f00cc990 0x7f91f00cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f5cf7640 1 -- 172.21.15.6:0/3665570232 >> v1:172.21.15.92:6789/0 conn(0x7f91f00d05d0 legacy=0x7f91f00d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.701+0000 7f91f5cf7640 1 -- 172.21.15.6:0/3665570232 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f00b2750 con 0x7f91f00d17f0 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.702+0000 7f91f5cf7640 1 --2- 172.21.15.6:0/3665570232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d17f0 0x7f91f00b17b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f91f006e680 tx=0x7f91ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbb9d2405a743b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.702+0000 7f91f4cf5640 1 -- 172.21.15.6:0/3665570232 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec034070 con 0x7f91f00d17f0 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.702+0000 7f91f4cf5640 1 -- 172.21.15.6:0/3665570232 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ec030860 con 0x7f91f00d17f0 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.702+0000 7f91f4cf5640 1 -- 172.21.15.6:0/3665570232 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec030c30 con 0x7f91f00d17f0 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 -- 172.21.15.6:0/3665570232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d17f0 msgr2=0x7f91f00b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/3665570232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d17f0 0x7f91f00b17b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f91f006e680 tx=0x7f91ec02b9a0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 -- 172.21.15.6:0/3665570232 shutdown_connections 2024-03-18T14:42:44.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/3665570232 >> v2:172.21.15.92:3300/0 conn(0x7f91f00cc990 0x7f91f00cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/3665570232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d17f0 0x7f91f00b17b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 -- 172.21.15.6:0/3665570232 >> 172.21.15.6:0/3665570232 conn(0x7f91f008b130 msgr2=0x7f91f008b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 -- 172.21.15.6:0/3665570232 shutdown_connections 2024-03-18T14:42:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.703+0000 7f91f7f5b640 1 -- 172.21.15.6:0/3665570232 wait complete. 2024-03-18T14:42:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.704+0000 7f91f7f5b640 1 Processor -- start 2024-03-18T14:42:44.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.704+0000 7f91f7f5b640 1 -- start start 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91f00cc990 0x7f91f00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d05d0 0x7f91f006de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91f00d17f0 0x7f91f0156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f91f00d0a30 con 0x7f91f00d05d0 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91f015c080 con 0x7f91f00d17f0 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f91f00cd340 con 0x7f91f00cc990 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d05d0 0x7f91f006de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d05d0 0x7f91f006de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50804/0 (socket says 172.21.15.6:50804) 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 -- 172.21.15.6:0/954819556 learned_addr learned my addr 172.21.15.6:0/954819556 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:44.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f5cf7640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91f00cc990 0x7f91f00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f64f8640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91f00d17f0 0x7f91f0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 -- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91f00cc990 msgr2=0x7f91f00c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91f00cc990 0x7f91f00c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 -- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91f00d17f0 msgr2=0x7f91f0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91f00d17f0 0x7f91f0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 -- 172.21.15.6:0/954819556 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f0156a40 con 0x7f91f00d05d0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f54f6640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d05d0 0x7f91f006de00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f91e000c400 tx=0x7f91e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91e67fc640 1 -- 172.21.15.6:0/954819556 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e00052d0 con 0x7f91f00d05d0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91e67fc640 1 -- 172.21.15.6:0/954819556 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91e0005490 con 0x7f91f00d05d0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f00cab30 con 0x7f91f00d05d0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91e67fc640 1 -- 172.21.15.6:0/954819556 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e0007260 con 0x7f91f00d05d0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.705+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f00cb0f0 con 0x7f91f00d05d0 2024-03-18T14:42:44.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.706+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f009fa80 con 0x7f91f00d05d0 2024-03-18T14:42:44.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.707+0000 7f91e67fc640 1 -- 172.21.15.6:0/954819556 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f91e0007400 con 0x7f91f00d05d0 2024-03-18T14:42:44.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.707+0000 7f91e67fc640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f91d406bbd0 0x7f91d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:44.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.707+0000 7f91e67fc640 1 -- 172.21.15.6:0/954819556 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f91e008a3d0 con 0x7f91f00d05d0 2024-03-18T14:42:44.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.708+0000 7f91f5cf7640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f91d406bbd0 0x7f91d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:44.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.708+0000 7f91f5cf7640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f91d406bbd0 0x7f91d406e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f91ec036000 tx=0x7f91ec03b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.713+0000 7f91e67fc640 1 -- 172.21.15.6:0/954819556 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f91e00583d0 con 0x7f91f00d05d0 2024-03-18T14:42:44.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.932+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91f00d1f80 con 0x7f91f00d05d0 2024-03-18T14:42:44.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.933+0000 7f91e67fc640 1 -- 172.21.15.6:0/954819556 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7f91e005c470 con 0x7f91f00d05d0 2024-03-18T14:42:44.935 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:44.935 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:42:44.935 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:42:44.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f91d406bbd0 msgr2=0x7f91d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f91d406bbd0 0x7f91d406e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f91ec036000 tx=0x7f91ec03b040 comp rx=0 tx=0).stop 2024-03-18T14:42:44.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d05d0 msgr2=0x7f91f006de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d05d0 0x7f91f006de00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f91e000c400 tx=0x7f91e000c8c0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 shutdown_connections 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f91d406bbd0 0x7f91d406e090 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91f00d17f0 0x7f91f0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00d05d0 0x7f91f006de00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 --2- 172.21.15.6:0/954819556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91f00cc990 0x7f91f00c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 >> 172.21.15.6:0/954819556 conn(0x7f91f008b130 msgr2=0x7f91f0070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 shutdown_connections 2024-03-18T14:42:44.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:44.934+0000 7f91f7f5b640 1 -- 172.21.15.6:0/954819556 wait complete. 2024-03-18T14:42:44.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:45.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.048+0000 7f2164cc6640 1 Processor -- start 2024-03-18T14:42:45.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.048+0000 7f2164cc6640 1 -- start start 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f2164cc6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f216015ab10 0x7f21600b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f2164cc6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f21600cb680 0x7f21600cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f2164cc6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21600cc000 con 0x7f21600ae840 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f2164cc6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f216006fca0 con 0x7f216015ab10 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f2164cc6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f21600cdb60 con 0x7f21600cb680 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215e59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f216015ab10 0x7f21600b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215e59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f216015ab10 0x7f21600b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50814/0 (socket says 172.21.15.6:50814) 2024-03-18T14:42:45.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215e59c640 1 -- 172.21.15.6:0/44537208 learned_addr learned my addr 172.21.15.6:0/44537208 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:45.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215ed9d640 1 --2- 172.21.15.6:0/44537208 >> v2:172.21.15.92:3300/0 conn(0x7f21600cb680 0x7f21600cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215e59c640 1 -- 172.21.15.6:0/44537208 >> v2:172.21.15.92:3300/0 conn(0x7f21600cb680 msgr2=0x7f21600cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:45.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215e59c640 1 --2- 172.21.15.6:0/44537208 >> v2:172.21.15.92:3300/0 conn(0x7f21600cb680 0x7f21600cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215e59c640 1 -- 172.21.15.6:0/44537208 >> v1:172.21.15.92:6789/0 conn(0x7f21600ae840 legacy=0x7f21600cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.049+0000 7f215e59c640 1 -- 172.21.15.6:0/44537208 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216006f120 con 0x7f216015ab10 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.050+0000 7f215e59c640 1 --2- 172.21.15.6:0/44537208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f21600b51b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f21600c68d0 tx=0x7f215002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a2f3165b7fd048 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.050+0000 7f215d59a640 1 -- 172.21.15.6:0/44537208 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215003b070 con 0x7f216015ab10 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.050+0000 7f215d59a640 1 -- 172.21.15.6:0/44537208 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2150036440 con 0x7f216015ab10 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.050+0000 7f215d59a640 1 -- 172.21.15.6:0/44537208 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150036810 con 0x7f216015ab10 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 -- 172.21.15.6:0/44537208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 msgr2=0x7f21600b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 --2- 172.21.15.6:0/44537208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f21600b51b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f21600c68d0 tx=0x7f215002e170 comp rx=0 tx=0).stop 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 -- 172.21.15.6:0/44537208 shutdown_connections 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 --2- 172.21.15.6:0/44537208 >> v2:172.21.15.92:3300/0 conn(0x7f21600cb680 0x7f21600cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 --2- 172.21.15.6:0/44537208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f21600b51b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 -- 172.21.15.6:0/44537208 >> 172.21.15.6:0/44537208 conn(0x7f21600b1470 msgr2=0x7f21600b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 -- 172.21.15.6:0/44537208 shutdown_connections 2024-03-18T14:42:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.051+0000 7f2164cc6640 1 -- 172.21.15.6:0/44537208 wait complete. 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.052+0000 7f2164cc6640 1 Processor -- start 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.052+0000 7f2164cc6640 1 -- start start 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f2164cc6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21600ae840 0x7f216015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f2164cc6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21600cb680 0x7f216015c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f2164cc6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f216015cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f2164cc6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f216006f9c0 con 0x7f216015ab10 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f2164cc6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f216006f6e0 con 0x7f21600cb680 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f2164cc6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f216015aef0 con 0x7f21600ae840 2024-03-18T14:42:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f216015cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f216015cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50826/0 (socket says 172.21.15.6:50826) 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 -- 172.21.15.6:0/1240468222 learned_addr learned my addr 172.21.15.6:0/1240468222 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215dd9b640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21600cb680 0x7f216015c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215e59c640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21600ae840 0x7f216015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 -- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21600ae840 msgr2=0x7f216015bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21600ae840 0x7f216015bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 -- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21600cb680 msgr2=0x7f216015c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21600cb680 0x7f216015c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.053+0000 7f215ed9d640 1 -- 172.21.15.6:0/1240468222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21600ae4f0 con 0x7f216015ab10 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.054+0000 7f215ed9d640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f216015cbb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f215800efc0 tx=0x7f215800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.054+0000 7f214effd640 1 -- 172.21.15.6:0/1240468222 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2158018020 con 0x7f216015ab10 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.054+0000 7f214effd640 1 -- 172.21.15.6:0/1240468222 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2158014480 con 0x7f216015ab10 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.054+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21600ad4b0 con 0x7f216015ab10 2024-03-18T14:42:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.054+0000 7f214effd640 1 -- 172.21.15.6:0/1240468222 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2158014870 con 0x7f216015ab10 2024-03-18T14:42:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.054+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216015b3e0 con 0x7f216015ab10 2024-03-18T14:42:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.055+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f216015b1f0 con 0x7f216015ab10 2024-03-18T14:42:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.055+0000 7f214effd640 1 -- 172.21.15.6:0/1240468222 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2158005d10 con 0x7f216015ab10 2024-03-18T14:42:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.056+0000 7f214effd640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f214006bc00 0x7f214006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.056+0000 7f214effd640 1 -- 172.21.15.6:0/1240468222 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f215808b7d0 con 0x7f216015ab10 2024-03-18T14:42:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.056+0000 7f215e59c640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f214006bc00 0x7f214006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.056+0000 7f215e59c640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f214006bc00 0x7f214006e0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2150038000 tx=0x7f2150008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.060+0000 7f214effd640 1 -- 172.21.15.6:0/1240468222 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2158022070 con 0x7f216015ab10 2024-03-18T14:42:45.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.281+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f216004f620 con 0x7f216015ab10 2024-03-18T14:42:45.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.282+0000 7f214effd640 1 -- 172.21.15.6:0/1240468222 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5168 (secure 0 0 0) 0x7f2158010040 con 0x7f216015ab10 2024-03-18T14:42:45.283 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:45.284 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/761154439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":761154439},{"type":"v1","addr":"172.21.15.6:6835","nonce":761154439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2313837523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2313837523},{"type":"v1","addr":"172.21.15.6:6834","nonce":2313837523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2084309992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2084309992},{"type":"v1","addr":"172.21.15.92:6836","nonce":2084309992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:41:34.883167+0000","modified":"2024-03-18T14:41:56.958188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6837/2327775013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":2327775013},{"type":"v1","addr":"172.21.15.92:6837","nonce":2327775013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-18T14:42:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f214006bc00 msgr2=0x7f214006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f214006bc00 0x7f214006e0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2150038000 tx=0x7f2150008040 comp rx=0 tx=0).stop 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 msgr2=0x7f216015cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f216015cbb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f215800efc0 tx=0x7f215800c490 comp rx=0 tx=0).stop 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 shutdown_connections 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f214006bc00 0x7f214006e0c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f216015ab10 0x7f216015cbb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21600cb680 0x7f216015c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 --2- 172.21.15.6:0/1240468222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21600ae840 0x7f216015bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 >> 172.21.15.6:0/1240468222 conn(0x7f21600b1470 msgr2=0x7f21600ccee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 shutdown_connections 2024-03-18T14:42:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.284+0000 7f2164cc6640 1 -- 172.21.15.6:0/1240468222 wait complete. 2024-03-18T14:42:45.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:45.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.396+0000 7efdde3c4640 1 Processor -- start 2024-03-18T14:42:45.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.396+0000 7efdde3c4640 1 -- start start 2024-03-18T14:42:45.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdde3c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b1c40 0x7efdd80d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdde3c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdd80cfd80 con 0x7efdd806cce0 2024-03-18T14:42:45.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdde3c4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdd80b2270 con 0x7efdd80b17b0 2024-03-18T14:42:45.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdde3c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efdd80b2df0 con 0x7efdd80b1c40 2024-03-18T14:42:45.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdd7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efdd806cce0 0x7efdd806d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52724/0 (socket says 172.21.15.6:52724) 2024-03-18T14:42:45.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdd7fff640 1 -- 172.21.15.6:0/532979993 learned_addr learned my addr 172.21.15.6:0/532979993 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efddc961640 1 --2- 172.21.15.6:0/532979993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b1c40 0x7efdd80d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdd6ffd640 1 -- 172.21.15.6:0/532979993 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 563024128 0 0) 0x7efdd80cfd80 con 0x7efdd806cce0 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdd6ffd640 1 -- 172.21.15.6:0/532979993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdc0003520 con 0x7efdd806cce0 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdd6ffd640 1 -- 172.21.15.6:0/532979993 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886857302 0 0) 0x7efdd80b2270 con 0x7efdd80b17b0 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efdd6ffd640 1 -- 172.21.15.6:0/532979993 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdd80cfd80 con 0x7efdd80b17b0 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efddc961640 1 -- 172.21.15.6:0/532979993 >> v1:172.21.15.92:6789/0 conn(0x7efdd80b17b0 legacy=0x7efdd80d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.397+0000 7efddc961640 1 -- 172.21.15.6:0/532979993 >> v1:172.21.15.6:6789/0 conn(0x7efdd806cce0 legacy=0x7efdd806d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.398+0000 7efddc961640 1 -- 172.21.15.6:0/532979993 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd80b2550 con 0x7efdd80b1c40 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.398+0000 7efddc961640 1 --2- 172.21.15.6:0/532979993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b1c40 0x7efdd80d0970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009dc0 tx=0x7efdd002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90b8b3c8d9663318 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.398+0000 7efdd6ffd640 1 -- 172.21.15.6:0/532979993 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0039070 con 0x7efdd80b1c40 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.398+0000 7efdd6ffd640 1 -- 172.21.15.6:0/532979993 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd002cc40 con 0x7efdd80b1c40 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.398+0000 7efdd6ffd640 1 -- 172.21.15.6:0/532979993 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0034440 con 0x7efdd80b1c40 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.399+0000 7efdde3c4640 1 -- 172.21.15.6:0/532979993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b1c40 msgr2=0x7efdd80d0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.399+0000 7efdde3c4640 1 --2- 172.21.15.6:0/532979993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b1c40 0x7efdd80d0970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009dc0 tx=0x7efdd002be90 comp rx=0 tx=0).stop 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.399+0000 7efdde3c4640 1 -- 172.21.15.6:0/532979993 shutdown_connections 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.399+0000 7efdde3c4640 1 --2- 172.21.15.6:0/532979993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b1c40 0x7efdd80d0970 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.399+0000 7efdde3c4640 1 -- 172.21.15.6:0/532979993 >> 172.21.15.6:0/532979993 conn(0x7efdd808b130 msgr2=0x7efdd808b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.399+0000 7efdde3c4640 1 -- 172.21.15.6:0/532979993 shutdown_connections 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.399+0000 7efdde3c4640 1 -- 172.21.15.6:0/532979993 wait complete. 2024-03-18T14:42:45.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.400+0000 7efdde3c4640 1 Processor -- start 2024-03-18T14:42:45.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.400+0000 7efdde3c4640 1 -- start start 2024-03-18T14:42:45.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdde3c4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdd806cce0 0x7efdd81f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdde3c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b17b0 0x7efdd8160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdde3c4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd80b1c40 0x7efdd8155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdde3c4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efdd815e330 con 0x7efdd806cce0 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdde3c4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efdd80b2b10 con 0x7efdd80b1c40 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdde3c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efdd806c690 con 0x7efdd80b17b0 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdd806cce0 0x7efdd81f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdd806cce0 0x7efdd81f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50828/0 (socket says 172.21.15.6:50828) 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd7fff640 1 -- 172.21.15.6:0/1194526924 learned_addr learned my addr 172.21.15.6:0/1194526924 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efddc961640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd80b1c40 0x7efdd8155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd77fe640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b17b0 0x7efdd8160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd7fff640 1 -- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b17b0 msgr2=0x7efdd8160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd7fff640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b17b0 0x7efdd8160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd7fff640 1 -- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd80b1c40 msgr2=0x7efdd8155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.401+0000 7efdd7fff640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd80b1c40 0x7efdd8155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.402+0000 7efdd7fff640 1 -- 172.21.15.6:0/1194526924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd81f7a30 con 0x7efdd806cce0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.402+0000 7efdd7fff640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdd806cce0 0x7efdd81f73a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00c530 tx=0x7efdcc00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.402+0000 7efdd4ff9640 1 -- 172.21.15.6:0/1194526924 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc017070 con 0x7efdd806cce0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.402+0000 7efdd4ff9640 1 -- 172.21.15.6:0/1194526924 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdcc005ce0 con 0x7efdd806cce0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.402+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd81f7d10 con 0x7efdd806cce0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.402+0000 7efdd4ff9640 1 -- 172.21.15.6:0/1194526924 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdcc007270 con 0x7efdd806cce0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.402+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd815e4b0 con 0x7efdd806cce0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.403+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd809fa80 con 0x7efdd806cce0 2024-03-18T14:42:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.403+0000 7efdd4ff9640 1 -- 172.21.15.6:0/1194526924 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efdcc01a070 con 0x7efdd806cce0 2024-03-18T14:42:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.404+0000 7efdd4ff9640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdc406bbd0 0x7efdc406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.404+0000 7efdd4ff9640 1 -- 172.21.15.6:0/1194526924 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7efdcc08a170 con 0x7efdd806cce0 2024-03-18T14:42:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.404+0000 7efdd77fe640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdc406bbd0 0x7efdc406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.404+0000 7efdd77fe640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdc406bbd0 0x7efdc406e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efdc8007fd0 tx=0x7efdc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.408+0000 7efdd4ff9640 1 -- 172.21.15.6:0/1194526924 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efdcc058220 con 0x7efdd806cce0 2024-03-18T14:42:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.593+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efdd80c5900 con 0x7efdd806cce0 2024-03-18T14:42:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.594+0000 7efdd4ff9640 1 -- 172.21.15.6:0/1194526924 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18807 (secure 0 0 0) 0x7efdcc05c2c0 con 0x7efdd806cce0 2024-03-18T14:42:45.596 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:45.596 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:16.655929+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:41:33.458302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T14:41:33.848049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T14:41:37.567980+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdc406bbd0 msgr2=0x7efdc406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdc406bbd0 0x7efdc406e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efdc8007fd0 tx=0x7efdc8009040 comp rx=0 tx=0).stop 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdd806cce0 msgr2=0x7efdd81f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdd806cce0 0x7efdd81f73a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00c530 tx=0x7efdcc00c9f0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 shutdown_connections 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdc406bbd0 0x7efdc406e090 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd80b1c40 0x7efdd8155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd80b17b0 0x7efdd8160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 --2- 172.21.15.6:0/1194526924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdd806cce0 0x7efdd81f73a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 >> 172.21.15.6:0/1194526924 conn(0x7efdd808b130 msgr2=0x7efdd80b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 shutdown_connections 2024-03-18T14:42:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.597+0000 7efdde3c4640 1 -- 172.21.15.6:0/1194526924 wait complete. 2024-03-18T14:42:45.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T14:42:45.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.711+0000 7f2870909640 1 Processor -- start 2024-03-18T14:42:45.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.711+0000 7f2870909640 1 -- start start 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f2870909640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f286c0d2300 0x7f286c0ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f2870909640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c0aeef0 con 0x7f286c15c090 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f2870909640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c0af1d0 con 0x7f286c15bcb0 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f2870909640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f286c0afb00 con 0x7f286c0d2300 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f286ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f286c0d2300 0x7f286c0ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f286ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f286c0d2300 0x7f286c0ae910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50836/0 (socket says 172.21.15.6:50836) 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f2869d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f286c15c090 0x7f286c0d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52728/0 (socket says 172.21.15.6:52728) 2024-03-18T14:42:45.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f286ad9d640 1 -- 172.21.15.6:0/4220176204 learned_addr learned my addr 172.21.15.6:0/4220176204 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:45.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.712+0000 7f286ad9d640 1 -- 172.21.15.6:0/4220176204 >> v1:172.21.15.92:6789/0 conn(0x7f286c15bcb0 legacy=0x7f286c0d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.713+0000 7f286ad9d640 1 -- 172.21.15.6:0/4220176204 >> v1:172.21.15.6:6789/0 conn(0x7f286c15c090 legacy=0x7f286c0d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.713+0000 7f286959a640 1 -- 172.21.15.6:0/4220176204 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3388846616 0 0) 0x7f286c0aeef0 con 0x7f286c15c090 2024-03-18T14:42:45.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.713+0000 7f286ad9d640 1 -- 172.21.15.6:0/4220176204 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c0af4b0 con 0x7f286c0d2300 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.713+0000 7f286ad9d640 1 --2- 172.21.15.6:0/4220176204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c0d2300 0x7f286c0ae910 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f285c009e50 tx=0x7f285c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63114538f96bdb57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.713+0000 7f286959a640 1 -- 172.21.15.6:0/4220176204 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c02cb20 con 0x7f286c0d2300 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.713+0000 7f286959a640 1 -- 172.21.15.6:0/4220176204 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285c02ccc0 con 0x7f286c0d2300 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.713+0000 7f286959a640 1 -- 172.21.15.6:0/4220176204 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c034610 con 0x7f286c0d2300 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.714+0000 7f2870909640 1 -- 172.21.15.6:0/4220176204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c0d2300 msgr2=0x7f286c0ae910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.714+0000 7f2870909640 1 --2- 172.21.15.6:0/4220176204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c0d2300 0x7f286c0ae910 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f285c009e50 tx=0x7f285c02bf80 comp rx=0 tx=0).stop 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.714+0000 7f2870909640 1 -- 172.21.15.6:0/4220176204 shutdown_connections 2024-03-18T14:42:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.714+0000 7f2870909640 1 --2- 172.21.15.6:0/4220176204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c0d2300 0x7f286c0ae910 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.714+0000 7f2870909640 1 -- 172.21.15.6:0/4220176204 >> 172.21.15.6:0/4220176204 conn(0x7f286c08b320 msgr2=0x7f286c08b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:45.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.714+0000 7f2870909640 1 -- 172.21.15.6:0/4220176204 shutdown_connections 2024-03-18T14:42:45.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.714+0000 7f2870909640 1 -- 172.21.15.6:0/4220176204 wait complete. 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.715+0000 7f2870909640 1 Processor -- start 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.715+0000 7f2870909640 1 -- start start 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2870909640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c0d2300 0x7f286c0c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2870909640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c15bcb0 0x7f286c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2870909640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c15c090 0x7f286c0b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2870909640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f286c0b1120 con 0x7f286c15bcb0 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2870909640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f286c0b0fa0 con 0x7f286c15c090 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2870909640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f286c0b12a0 con 0x7f286c0d2300 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2869d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c15bcb0 0x7f286c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2869d9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c15bcb0 0x7f286c0b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50852/0 (socket says 172.21.15.6:50852) 2024-03-18T14:42:45.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2869d9b640 1 -- 172.21.15.6:0/2311586416 learned_addr learned my addr 172.21.15.6:0/2311586416 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f286ad9d640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c15c090 0x7f286c0b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f286a59c640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c0d2300 0x7f286c0c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2869d9b640 1 -- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c0d2300 msgr2=0x7f286c0c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2869d9b640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c0d2300 0x7f286c0c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.716+0000 7f2869d9b640 1 -- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c15c090 msgr2=0x7f286c0b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2869d9b640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c15c090 0x7f286c0b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2869d9b640 1 -- 172.21.15.6:0/2311586416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c0b4eb0 con 0x7f286c15bcb0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2869d9b640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c15bcb0 0x7f286c0b4970 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f285400c490 tx=0x7f285400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2862ffd640 1 -- 172.21.15.6:0/2311586416 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854018070 con 0x7f286c15bcb0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2862ffd640 1 -- 172.21.15.6:0/2311586416 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28540074e0 con 0x7f286c15bcb0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286c0caf20 con 0x7f286c15bcb0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2862ffd640 1 -- 172.21.15.6:0/2311586416 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854004a20 con 0x7f286c15bcb0 2024-03-18T14:42:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.717+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f286c0ca960 con 0x7f286c15bcb0 2024-03-18T14:42:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.718+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286c04f5b0 con 0x7f286c15bcb0 2024-03-18T14:42:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.718+0000 7f2862ffd640 1 -- 172.21.15.6:0/2311586416 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f285401b070 con 0x7f286c15bcb0 2024-03-18T14:42:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.719+0000 7f2862ffd640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f284c06bbd0 0x7f284c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.719+0000 7f2862ffd640 1 -- 172.21.15.6:0/2311586416 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2854089fa0 con 0x7f286c15bcb0 2024-03-18T14:42:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.719+0000 7f286a59c640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f284c06bbd0 0x7f284c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.719+0000 7f286a59c640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f284c06bbd0 0x7f284c06e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f286c0cb250 tx=0x7f2864009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.723+0000 7f2862ffd640 1 -- 172.21.15.6:0/2311586416 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f28540580d0 con 0x7f286c15bcb0 2024-03-18T14:42:45.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.947+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f286c0cbee0 con 0x7f286c15bcb0 2024-03-18T14:42:45.948 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-18T14:42:45.947+0000 7f3765211640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T14:42:45.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.951+0000 7f2862ffd640 1 -- 172.21.15.6:0/2311586416 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f285405c170 con 0x7f286c15bcb0 2024-03-18T14:42:45.952 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T14:42:45.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f284c06bbd0 msgr2=0x7f284c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f284c06bbd0 0x7f284c06e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f286c0cb250 tx=0x7f2864009040 comp rx=0 tx=0).stop 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c15bcb0 msgr2=0x7f286c0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c15bcb0 0x7f286c0b4970 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f285400c490 tx=0x7f285400c950 comp rx=0 tx=0).stop 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 shutdown_connections 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f284c06bbd0 0x7f284c06e090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c15c090 0x7f286c0b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c15bcb0 0x7f286c0b4970 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 --2- 172.21.15.6:0/2311586416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c0d2300 0x7f286c0c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 >> 172.21.15.6:0/2311586416 conn(0x7f286c08b320 msgr2=0x7f286c0711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 shutdown_connections 2024-03-18T14:42:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:45.953+0000 7f2870909640 1 -- 172.21.15.6:0/2311586416 wait complete. 2024-03-18T14:42:45.958 INFO:tasks.ceph.mds.d.smithi092.stderr: -4672> 2024-03-18T14:41:39.954+0000 7fb870ff2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T14:42:45.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T14:42:45.987 INFO:tasks.ceph.mds.d.smithi092.stdout:starting mds.d at 2024-03-18T14:42:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.069+0000 7efdeae6c640 1 Processor -- start 2024-03-18T14:42:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.069+0000 7efdeae6c640 1 -- start start 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efdeae6c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efdeae6c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efde40b9940 0x7efde415e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efdeae6c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efde40cf290 con 0x7efde415e950 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efdeae6c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7efde40d21a0 con 0x7efde40b9940 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efdeae6c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efde415edd0 con 0x7efde40ce970 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efde8c08640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efde3fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efde40b9940 0x7efde415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efde9409640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efde415e950 0x7efde40b5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40536/0 (socket says 172.21.15.6:40536) 2024-03-18T14:42:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efde8c08640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55134/0 (socket says 172.21.15.6:55134) 2024-03-18T14:42:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.070+0000 7efde9409640 1 -- 172.21.15.6:0/3140236773 learned_addr learned my addr 172.21.15.6:0/3140236773 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.071+0000 7efde3fff640 1 -- 172.21.15.6:0/3140236773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 msgr2=0x7efde40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.071+0000 7efde3fff640 1 --2- 172.21.15.6:0/3140236773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.071+0000 7efde3fff640 1 -- 172.21.15.6:0/3140236773 >> v1:172.21.15.92:6789/0 conn(0x7efde415e950 legacy=0x7efde40b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.071+0000 7efde3fff640 1 -- 172.21.15.6:0/3140236773 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde415f0b0 con 0x7efde40b9940 2024-03-18T14:42:46.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.071+0000 7efde3fff640 1 --2- 172.21.15.6:0/3140236773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 0x7efde415e410 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efdd80097a0 tx=0x7efdd802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e12f8fec6e89257 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:46.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.071+0000 7efde37fe640 1 -- 172.21.15.6:0/3140236773 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8037070 con 0x7efde40b9940 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efde37fe640 1 -- 172.21.15.6:0/3140236773 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd8030710 con 0x7efde40b9940 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efde37fe640 1 -- 172.21.15.6:0/3140236773 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8030ae0 con 0x7efde40b9940 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 -- 172.21.15.6:0/3140236773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 msgr2=0x7efde415e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 --2- 172.21.15.6:0/3140236773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 0x7efde415e410 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efdd80097a0 tx=0x7efdd802b710 comp rx=0 tx=0).stop 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 -- 172.21.15.6:0/3140236773 shutdown_connections 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 --2- 172.21.15.6:0/3140236773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 0x7efde415e410 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 --2- 172.21.15.6:0/3140236773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 -- 172.21.15.6:0/3140236773 >> 172.21.15.6:0/3140236773 conn(0x7efde406c900 msgr2=0x7efde406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 -- 172.21.15.6:0/3140236773 shutdown_connections 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.072+0000 7efdeae6c640 1 -- 172.21.15.6:0/3140236773 wait complete. 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.073+0000 7efdeae6c640 1 Processor -- start 2024-03-18T14:42:46.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.073+0000 7efdeae6c640 1 -- start start 2024-03-18T14:42:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efdeae6c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 0x7efde4154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efdeae6c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efdeae6c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efde415e950 0x7efde40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efdeae6c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efde415bd70 con 0x7efde415e950 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efdeae6c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efde415f390 con 0x7efde40b9940 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efdeae6c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efde4154fd0 con 0x7efde40ce970 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efde9409640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efde415e950 0x7efde40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efde9409640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efde415e950 0x7efde40b4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50854/0 (socket says 172.21.15.6:50854) 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efde9409640 1 -- 172.21.15.6:0/734695316 learned_addr learned my addr 172.21.15.6:0/734695316 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efde3fff640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.074+0000 7efde8c08640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 0x7efde4154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde9409640 1 -- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 msgr2=0x7efde40b48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde9409640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40b48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde9409640 1 -- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 msgr2=0x7efde4154a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde9409640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 0x7efde4154a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde9409640 1 -- 172.21.15.6:0/734695316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde40c9880 con 0x7efde415e950 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde3fff640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde9409640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efde415e950 0x7efde40b4e10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efdd400c7b0 tx=0x7efdd400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde17fa640 1 -- 172.21.15.6:0/734695316 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4018070 con 0x7efde415e950 2024-03-18T14:42:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde17fa640 1 -- 172.21.15.6:0/734695316 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd40072d0 con 0x7efde415e950 2024-03-18T14:42:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde40cd190 con 0x7efde415e950 2024-03-18T14:42:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efde17fa640 1 -- 172.21.15.6:0/734695316 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd40047e0 con 0x7efde415e950 2024-03-18T14:42:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.075+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde40c7dd0 con 0x7efde415e950 2024-03-18T14:42:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.076+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde40751b0 con 0x7efde415e950 2024-03-18T14:42:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.076+0000 7efde17fa640 1 -- 172.21.15.6:0/734695316 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efdd401b070 con 0x7efde415e950 2024-03-18T14:42:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.077+0000 7efde17fa640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdd006bbd0 0x7efdd006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.077+0000 7efde17fa640 1 -- 172.21.15.6:0/734695316 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7efdd408a1d0 con 0x7efde415e950 2024-03-18T14:42:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.077+0000 7efde8c08640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdd006bbd0 0x7efdd006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.078+0000 7efde8c08640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdd006bbd0 0x7efdd006e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efddc0113e0 tx=0x7efddc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:46.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.081+0000 7efde17fa640 1 -- 172.21.15.6:0/734695316 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efdd40582a0 con 0x7efde415e950 2024-03-18T14:42:46.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.305+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7efde40c7a80 con 0x7efde415e950 2024-03-18T14:42:46.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.954+0000 7efde17fa640 1 -- 172.21.15.6:0/734695316 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7efdd405c340 con 0x7efde415e950 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdd006bbd0 msgr2=0x7efdd006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdd006bbd0 0x7efdd006e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efddc0113e0 tx=0x7efddc00b040 comp rx=0 tx=0).stop 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efde415e950 msgr2=0x7efde40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efde415e950 0x7efde40b4e10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efdd400c7b0 tx=0x7efdd400cc70 comp rx=0 tx=0).stop 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 shutdown_connections 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efdd006bbd0 0x7efdd006e090 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efde415e950 0x7efde40b4e10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efde40ce970 0x7efde40b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 --2- 172.21.15.6:0/734695316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efde40b9940 0x7efde4154a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 >> 172.21.15.6:0/734695316 conn(0x7efde406c900 msgr2=0x7efde40d2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 shutdown_connections 2024-03-18T14:42:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:46.957+0000 7efdeae6c640 1 -- 172.21.15.6:0/734695316 wait complete. 2024-03-18T14:42:46.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T14:42:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.083+0000 7fc67da97640 1 Processor -- start 2024-03-18T14:42:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.083+0000 7fc67da97640 1 -- start start 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.083+0000 7fc67da97640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6780d1de0 0x7fc6780d3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.083+0000 7fc67da97640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6780d22c0 con 0x7fc67806c730 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.083+0000 7fc67da97640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6780d25a0 con 0x7fc67815e0b0 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.083+0000 7fc67da97640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc6780d2880 con 0x7fc6780d1de0 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc676ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6780d1de0 0x7fc6780d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc677fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc67806c730 0x7fc6780b6d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52738/0 (socket says 172.21.15.6:52738) 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc676ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6780d1de0 0x7fc6780d3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50868/0 (socket says 172.21.15.6:50868) 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc677fff640 1 -- 172.21.15.6:0/1031416806 learned_addr learned my addr 172.21.15.6:0/1031416806 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc676ffd640 1 -- 172.21.15.6:0/1031416806 >> v1:172.21.15.92:6789/0 conn(0x7fc67815e0b0 legacy=0x7fc6780b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:47.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc676ffd640 1 -- 172.21.15.6:0/1031416806 >> v1:172.21.15.6:6789/0 conn(0x7fc67806c730 legacy=0x7fc6780b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:47.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc676ffd640 1 -- 172.21.15.6:0/1031416806 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6780d2b60 con 0x7fc6780d1de0 2024-03-18T14:42:47.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.084+0000 7fc6767fc640 1 -- 172.21.15.6:0/1031416806 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1521792860 0 0) 0x7fc6780d22c0 con 0x7fc67806c730 2024-03-18T14:42:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc676ffd640 1 --2- 172.21.15.6:0/1031416806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6780d1de0 0x7fc6780d3360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6680097a0 tx=0x7fc66802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd773142c2cdac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc6767fc640 1 -- 172.21.15.6:0/1031416806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66802f810 con 0x7fc6780d1de0 2024-03-18T14:42:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc6767fc640 1 -- 172.21.15.6:0/1031416806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc66802f9b0 con 0x7fc6780d1de0 2024-03-18T14:42:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc6767fc640 1 -- 172.21.15.6:0/1031416806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66802fd80 con 0x7fc6780d1de0 2024-03-18T14:42:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc67da97640 1 -- 172.21.15.6:0/1031416806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6780d1de0 msgr2=0x7fc6780d3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc67da97640 1 --2- 172.21.15.6:0/1031416806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6780d1de0 0x7fc6780d3360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6680097a0 tx=0x7fc66802b9f0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc67da97640 1 -- 172.21.15.6:0/1031416806 shutdown_connections 2024-03-18T14:42:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc67da97640 1 --2- 172.21.15.6:0/1031416806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6780d1de0 0x7fc6780d3360 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc67da97640 1 -- 172.21.15.6:0/1031416806 >> 172.21.15.6:0/1031416806 conn(0x7fc678072ce0 msgr2=0x7fc6780730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc67da97640 1 -- 172.21.15.6:0/1031416806 shutdown_connections 2024-03-18T14:42:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.085+0000 7fc67da97640 1 -- 172.21.15.6:0/1031416806 wait complete. 2024-03-18T14:42:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 Processor -- start 2024-03-18T14:42:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 -- start start 2024-03-18T14:42:47.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc67806c730 0x7fc6781f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6780d1de0 0x7fc6780cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc67815e0b0 0x7fc6781f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc6780d38a0 con 0x7fc67815e0b0 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc6780b7410 con 0x7fc67806c730 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.087+0000 7fc67da97640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc6781f42b0 con 0x7fc6780d1de0 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc67815e0b0 0x7fc6781f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc67815e0b0 0x7fc6781f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50882/0 (socket says 172.21.15.6:50882) 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 -- 172.21.15.6:0/3657902396 learned_addr learned my addr 172.21.15.6:0/3657902396 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc6777fe640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc67806c730 0x7fc6781f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc676ffd640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6780d1de0 0x7fc6780cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 -- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6780d1de0 msgr2=0x7fc6780cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6780d1de0 0x7fc6780cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 -- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc67806c730 msgr2=0x7fc6781f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc67806c730 0x7fc6781f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 -- 172.21.15.6:0/3657902396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67815e530 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc677fff640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc67815e0b0 0x7fc6781f6a00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc67000c7a0 tx=0x7fc67000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc66ffff640 1 -- 172.21.15.6:0/3657902396 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670018070 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc66ffff640 1 -- 172.21.15.6:0/3657902396 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67000a640 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc66ffff640 1 -- 172.21.15.6:0/3657902396 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67000aab0 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67815eab0 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.088+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6780cbf90 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.089+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc67808afe0 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.090+0000 7fc66ffff640 1 -- 172.21.15.6:0/3657902396 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc67001b070 con 0x7fc67815e0b0 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.090+0000 7fc66ffff640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc65806bbd0 0x7fc65806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.090+0000 7fc66ffff640 1 -- 172.21.15.6:0/3657902396 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7fc67008a440 con 0x7fc67815e0b0 2024-03-18T14:42:47.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.090+0000 7fc6777fe640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc65806bbd0 0x7fc65806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:47.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.091+0000 7fc6777fe640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc65806bbd0 0x7fc65806e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc6780b6240 tx=0x7fc660009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.095+0000 7fc66ffff640 1 -- 172.21.15.6:0/3657902396 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc670058480 con 0x7fc67815e0b0 2024-03-18T14:42:47.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.276+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc67804f620 con 0x7fc67815e0b0 2024-03-18T14:42:47.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.959+0000 7fc66ffff640 1 -- 172.21.15.6:0/3657902396 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7fc67005c520 con 0x7fc67815e0b0 2024-03-18T14:42:47.961 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc65806bbd0 msgr2=0x7fc65806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc65806bbd0 0x7fc65806e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc6780b6240 tx=0x7fc660009040 comp rx=0 tx=0).stop 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc67815e0b0 msgr2=0x7fc6781f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc67815e0b0 0x7fc6781f6a00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc67000c7a0 tx=0x7fc67000cc60 comp rx=0 tx=0).stop 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 shutdown_connections 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc65806bbd0 0x7fc65806e090 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc67815e0b0 0x7fc6781f6a00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6780d1de0 0x7fc6780cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 --2- 172.21.15.6:0/3657902396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc67806c730 0x7fc6781f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 >> 172.21.15.6:0/3657902396 conn(0x7fc678072ce0 msgr2=0x7fc6780b77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 shutdown_connections 2024-03-18T14:42:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:47.962+0000 7fc67da97640 1 -- 172.21.15.6:0/3657902396 wait complete. 2024-03-18T14:42:47.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T14:42:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.086+0000 7f714fcf7640 1 Processor -- start 2024-03-18T14:42:48.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.086+0000 7f714fcf7640 1 -- start start 2024-03-18T14:42:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.086+0000 7f714fcf7640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f71480cc5d0 0x7f71480ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.086+0000 7f714fcf7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f71480b1b50 0x7f71480cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714fcf7640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71480cce00 con 0x7f714806b1c0 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714fcf7640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f71480b22c0 con 0x7f71480b1b50 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714fcf7640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f71480cd0e0 con 0x7f71480cc5d0 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714d292640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f71480b1b50 0x7f71480cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714d292640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f71480b1b50 0x7f71480cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50886/0 (socket says 172.21.15.6:50886) 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714d292640 1 -- 172.21.15.6:0/1437892385 learned_addr learned my addr 172.21.15.6:0/1437892385 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714da93640 1 --2- 172.21.15.6:0/1437892385 >> v2:172.21.15.92:3300/0 conn(0x7f71480cc5d0 0x7f71480ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714d292640 1 -- 172.21.15.6:0/1437892385 >> v2:172.21.15.92:3300/0 conn(0x7f71480cc5d0 msgr2=0x7f71480ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714d292640 1 --2- 172.21.15.6:0/1437892385 >> v2:172.21.15.92:3300/0 conn(0x7f71480cc5d0 0x7f71480ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714d292640 1 -- 172.21.15.6:0/1437892385 >> v1:172.21.15.92:6789/0 conn(0x7f714806b1c0 legacy=0x7f71480cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.087+0000 7f714d292640 1 -- 172.21.15.6:0/1437892385 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71480cd3c0 con 0x7f71480b1b50 2024-03-18T14:42:48.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.088+0000 7f714d292640 1 --2- 172.21.15.6:0/1437892385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480cdbc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7148071470 tx=0x7f713802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bbb0cd13bbc344 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.088+0000 7f714ca91640 1 -- 172.21.15.6:0/1437892385 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138034070 con 0x7f71480b1b50 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.088+0000 7f714ca91640 1 -- 172.21.15.6:0/1437892385 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7138030860 con 0x7f71480b1b50 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.088+0000 7f714ca91640 1 -- 172.21.15.6:0/1437892385 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138030c30 con 0x7f71480b1b50 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.088+0000 7f714fcf7640 1 -- 172.21.15.6:0/1437892385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 msgr2=0x7f71480cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.088+0000 7f714fcf7640 1 --2- 172.21.15.6:0/1437892385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480cdbc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7148071470 tx=0x7f713802b9a0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.089+0000 7f714fcf7640 1 -- 172.21.15.6:0/1437892385 shutdown_connections 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.089+0000 7f714fcf7640 1 --2- 172.21.15.6:0/1437892385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480cdbc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.089+0000 7f714fcf7640 1 --2- 172.21.15.6:0/1437892385 >> v2:172.21.15.92:3300/0 conn(0x7f71480cc5d0 0x7f71480ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.089+0000 7f714fcf7640 1 -- 172.21.15.6:0/1437892385 >> 172.21.15.6:0/1437892385 conn(0x7f71480d0980 msgr2=0x7f71480d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.089+0000 7f714fcf7640 1 -- 172.21.15.6:0/1437892385 shutdown_connections 2024-03-18T14:42:48.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.089+0000 7f714fcf7640 1 -- 172.21.15.6:0/1437892385 wait complete. 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.090+0000 7f714fcf7640 1 Processor -- start 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.090+0000 7f714fcf7640 1 -- start start 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.090+0000 7f714fcf7640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f714806b1c0 0x7f714809a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.090+0000 7f714fcf7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714fcf7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71480cc5d0 0x7f71480b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714fcf7640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f71480ce100 con 0x7f71480b1b50 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714fcf7640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f71480b1f90 con 0x7f71480cc5d0 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714fcf7640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f71480b5390 con 0x7f714806b1c0 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:48.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480b4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50896/0 (socket says 172.21.15.6:50896) 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 -- 172.21.15.6:0/2844928047 learned_addr learned my addr 172.21.15.6:0/2844928047 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714e294640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71480cc5d0 0x7f71480b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714da93640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f714806b1c0 0x7f714809a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 -- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f714806b1c0 msgr2=0x7f714809a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f714806b1c0 0x7f714809a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 -- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71480cc5d0 msgr2=0x7f71480b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71480cc5d0 0x7f71480b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 -- 172.21.15.6:0/2844928047 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71480b5be0 con 0x7f71480b1b50 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714e294640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71480cc5d0 0x7f71480b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714d292640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480b4910 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f713802b9e0 tx=0x7f7138004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f713e7fc640 1 -- 172.21.15.6:0/2844928047 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138048070 con 0x7f71480b1b50 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f713e7fc640 1 -- 172.21.15.6:0/2844928047 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7138030660 con 0x7f71480b1b50 2024-03-18T14:42:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71480b5620 con 0x7f71480b1b50 2024-03-18T14:42:48.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f713e7fc640 1 -- 172.21.15.6:0/2844928047 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138034040 con 0x7f71480b1b50 2024-03-18T14:42:48.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.091+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f714809cef0 con 0x7f71480b1b50 2024-03-18T14:42:48.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.093+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f714809e970 con 0x7f71480b1b50 2024-03-18T14:42:48.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.093+0000 7f713e7fc640 1 -- 172.21.15.6:0/2844928047 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f713804b070 con 0x7f71480b1b50 2024-03-18T14:42:48.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.093+0000 7f713e7fc640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f712c06bbd0 0x7f712c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:48.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.093+0000 7f713e7fc640 1 -- 172.21.15.6:0/2844928047 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6383+0+0 (secure 0 0 0) 0x7f71380acd50 con 0x7f71480b1b50 2024-03-18T14:42:48.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.094+0000 7f714da93640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f712c06bbd0 0x7f712c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:48.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.094+0000 7f714da93640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f712c06bbd0 0x7f712c06e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7144006fd0 tx=0x7f7144008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:48.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.098+0000 7f713e7fc640 1 -- 172.21.15.6:0/2844928047 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f71380b0030 con 0x7f71480b1b50 2024-03-18T14:42:48.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.281+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f714804f620 con 0x7f71480b1b50 2024-03-18T14:42:48.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.964+0000 7f713e7fc640 1 -- 172.21.15.6:0/2844928047 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7f713807b5a0 con 0x7f71480b1b50 2024-03-18T14:42:48.966 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-18T14:42:48.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f712c06bbd0 msgr2=0x7f712c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:48.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f712c06bbd0 0x7f712c06e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7144006fd0 tx=0x7f7144008040 comp rx=0 tx=0).stop 2024-03-18T14:42:48.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 msgr2=0x7f71480b4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:48.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480b4910 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f713802b9e0 tx=0x7f7138004730 comp rx=0 tx=0).stop 2024-03-18T14:42:48.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 shutdown_connections 2024-03-18T14:42:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f712c06bbd0 0x7f712c06e090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71480cc5d0 0x7f71480b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71480b1b50 0x7f71480b4910 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 --2- 172.21.15.6:0/2844928047 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f714806b1c0 0x7f714809a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.968+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 >> 172.21.15.6:0/2844928047 conn(0x7f71480d0980 msgr2=0x7f71480ad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.969+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 shutdown_connections 2024-03-18T14:42:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:48.969+0000 7f714fcf7640 1 -- 172.21.15.6:0/2844928047 wait complete. 2024-03-18T14:42:48.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-18T14:42:49.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.102+0000 7fd422f3b640 1 Processor -- start 2024-03-18T14:42:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.102+0000 7fd422f3b640 1 -- start start 2024-03-18T14:42:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd422f3b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd41c0d1de0 0x7fd41c0d3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd422f3b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd41c06c730 0x7fd41c06cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd422f3b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd41c0d22c0 con 0x7fd41c15e0b0 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd422f3b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd41c06d0a0 con 0x7fd41c0d1de0 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd422f3b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd41c0d25a0 con 0x7fd41c06c730 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41bfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd41c0d1de0 0x7fd41c0d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41bfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd41c0d1de0 0x7fd41c0d3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50902/0 (socket says 172.21.15.6:50902) 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41bfff640 1 -- 172.21.15.6:0/2600359012 learned_addr learned my addr 172.21.15.6:0/2600359012 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd420cd7640 1 --1- 172.21.15.6:0/2600359012 >> v1:172.21.15.6:6789/0 conn(0x7fd41c15e0b0 0x7fd41c0b95f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52746/0 (socket says 172.21.15.6:52746) 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd4214d8640 1 --2- 172.21.15.6:0/2600359012 >> v2:172.21.15.92:3300/0 conn(0x7fd41c06c730 0x7fd41c06cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41bfff640 1 -- 172.21.15.6:0/2600359012 >> v2:172.21.15.92:3300/0 conn(0x7fd41c06c730 msgr2=0x7fd41c06cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41bfff640 1 --2- 172.21.15.6:0/2600359012 >> v2:172.21.15.92:3300/0 conn(0x7fd41c06c730 0x7fd41c06cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41bfff640 1 -- 172.21.15.6:0/2600359012 >> v1:172.21.15.6:6789/0 conn(0x7fd41c15e0b0 legacy=0x7fd41c0b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41bfff640 1 -- 172.21.15.6:0/2600359012 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c0d2880 con 0x7fd41c0d1de0 2024-03-18T14:42:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd41b7fe640 1 -- 172.21.15.6:0/2600359012 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 48848684 0 0) 0x7fd41c0d22c0 con 0x7fd41c15e0b0 2024-03-18T14:42:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.103+0000 7fd4214d8640 1 --2- 172.21.15.6:0/2600359012 >> v2:172.21.15.92:3300/0 conn(0x7fd41c06c730 0x7fd41c06cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.104+0000 7fd41bfff640 1 --2- 172.21.15.6:0/2600359012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c0d1de0 0x7fd41c0d3360 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0d1250 tx=0x7fd40802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a8c2cb6a777bad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.104+0000 7fd41b7fe640 1 -- 172.21.15.6:0/2600359012 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40802f850 con 0x7fd41c0d1de0 2024-03-18T14:42:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.104+0000 7fd41b7fe640 1 -- 172.21.15.6:0/2600359012 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd40802f9f0 con 0x7fd41c0d1de0 2024-03-18T14:42:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.104+0000 7fd41b7fe640 1 -- 172.21.15.6:0/2600359012 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40802fdc0 con 0x7fd41c0d1de0 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 -- 172.21.15.6:0/2600359012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c0d1de0 msgr2=0x7fd41c0d3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 --2- 172.21.15.6:0/2600359012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c0d1de0 0x7fd41c0d3360 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0d1250 tx=0x7fd40802ba30 comp rx=0 tx=0).stop 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 -- 172.21.15.6:0/2600359012 shutdown_connections 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 --2- 172.21.15.6:0/2600359012 >> v2:172.21.15.92:3300/0 conn(0x7fd41c06c730 0x7fd41c06cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 --2- 172.21.15.6:0/2600359012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c0d1de0 0x7fd41c0d3360 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 -- 172.21.15.6:0/2600359012 >> 172.21.15.6:0/2600359012 conn(0x7fd41c072ce0 msgr2=0x7fd41c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 -- 172.21.15.6:0/2600359012 shutdown_connections 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.105+0000 7fd422f3b640 1 -- 172.21.15.6:0/2600359012 wait complete. 2024-03-18T14:42:49.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.106+0000 7fd422f3b640 1 Processor -- start 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.106+0000 7fd422f3b640 1 -- start start 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd422f3b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd41c06c730 0x7fd41c152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd422f3b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd41c0d1de0 0x7fd41c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd422f3b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c15e0b0 0x7fd41c15fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd422f3b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c0d38a0 con 0x7fd41c15e0b0 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd422f3b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c0d2e40 con 0x7fd41c06c730 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd422f3b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd41c153220 con 0x7fd41c0d1de0 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c15e0b0 0x7fd41c15fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c15e0b0 0x7fd41c15fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50916/0 (socket says 172.21.15.6:50916) 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 -- 172.21.15.6:0/1135943806 learned_addr learned my addr 172.21.15.6:0/1135943806 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd420cd7640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd41c06c730 0x7fd41c152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd41bfff640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd41c0d1de0 0x7fd41c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:49.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 -- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd41c0d1de0 msgr2=0x7fd41c0b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd41c0d1de0 0x7fd41c0b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 -- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd41c06c730 msgr2=0x7fd41c152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd41c06c730 0x7fd41c152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 -- 172.21.15.6:0/1135943806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c0cb280 con 0x7fd41c15e0b0 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd420cd7640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd41c06c730 0x7fd41c152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.107+0000 7fd4214d8640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c15e0b0 0x7fd41c15fe00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd41400cef0 tx=0x7fd41400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.108+0000 7fd4197fa640 1 -- 172.21.15.6:0/1135943806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41401a070 con 0x7fd41c15e0b0 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.108+0000 7fd4197fa640 1 -- 172.21.15.6:0/1135943806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41400d860 con 0x7fd41c15e0b0 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.108+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41c0cb560 con 0x7fd41c15e0b0 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.108+0000 7fd4197fa640 1 -- 172.21.15.6:0/1135943806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41400dc50 con 0x7fd41c15e0b0 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.108+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41c0cbb20 con 0x7fd41c15e0b0 2024-03-18T14:42:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.109+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd41c08afe0 con 0x7fd41c15e0b0 2024-03-18T14:42:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.109+0000 7fd4197fa640 1 -- 172.21.15.6:0/1135943806 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd41401d070 con 0x7fd41c15e0b0 2024-03-18T14:42:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.110+0000 7fd4197fa640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd40006bbd0 0x7fd40006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.110+0000 7fd4197fa640 1 -- 172.21.15.6:0/1135943806 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7fd41408b2e0 con 0x7fd41c15e0b0 2024-03-18T14:42:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.110+0000 7fd420cd7640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd40006bbd0 0x7fd40006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.110+0000 7fd420cd7640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd40006bbd0 0x7fd40006e090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd410000f60 tx=0x7fd410009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.114+0000 7fd4197fa640 1 -- 172.21.15.6:0/1135943806 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd414059e90 con 0x7fd41c15e0b0 2024-03-18T14:42:49.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.297+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd41c04f620 con 0x7fd41c15e0b0 2024-03-18T14:42:49.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.969+0000 7fd4197fa640 1 -- 172.21.15.6:0/1135943806 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7fd414059790 con 0x7fd41c15e0b0 2024-03-18T14:42:49.971 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data_ec' removed 2024-03-18T14:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd40006bbd0 msgr2=0x7fd40006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd40006bbd0 0x7fd40006e090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd410000f60 tx=0x7fd410009040 comp rx=0 tx=0).stop 2024-03-18T14:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c15e0b0 msgr2=0x7fd41c15fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c15e0b0 0x7fd41c15fe00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd41400cef0 tx=0x7fd41400a710 comp rx=0 tx=0).stop 2024-03-18T14:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 shutdown_connections 2024-03-18T14:42:49.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd40006bbd0 0x7fd40006e090 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd41c15e0b0 0x7fd41c15fe00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd41c0d1de0 0x7fd41c0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 --2- 172.21.15.6:0/1135943806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd41c06c730 0x7fd41c152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:49.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.974+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 >> 172.21.15.6:0/1135943806 conn(0x7fd41c072ce0 msgr2=0x7fd41c0c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:49.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.975+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 shutdown_connections 2024-03-18T14:42:49.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:49.975+0000 7fd422f3b640 1 -- 172.21.15.6:0/1135943806 wait complete. 2024-03-18T14:42:49.991 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T14:42:49.991 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T14:42:49.991 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T14:42:49.991 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:49.991 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T14:42:49.991 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T14:42:49.991 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T14:42:49.992 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:49.992 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T14:42:49.992 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T14:42:49.992 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T14:42:49.992 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:49.992 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T14:42:49.992 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T14:42:49.992 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T14:42:49.993 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:42:50.001 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T14:42:50.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T14:42:50.002 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T14:42:50.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T14:42:50.004 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T14:42:50.004 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T14:42:50.005 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T14:42:50.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T14:42:50.044 INFO:tasks.ceph.mds.a:Started 2024-03-18T14:42:50.044 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:50.044 INFO:tasks.ceph.mds.c:Started 2024-03-18T14:42:50.044 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:50.047 INFO:tasks.ceph.mds.b:Started 2024-03-18T14:42:50.047 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:50.047 INFO:tasks.ceph.mds.d:Started 2024-03-18T14:42:50.047 DEBUG:teuthology.parallel:result is None 2024-03-18T14:42:50.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T14:42:50.122 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-18T14:42:50.123 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-18T14:42:50.133 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2024-03-18T14:42:50.133 INFO:tasks.ceph.mds.d.smithi092.stdout:starting mds.d at 2024-03-18T14:42:50.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.156+0000 7f67d20bf640 1 Processor -- start 2024-03-18T14:42:50.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.156+0000 7f67d20bf640 1 -- start start 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.156+0000 7f67d20bf640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f67cc06c7a0 0x7f67cc0d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.156+0000 7f67d20bf640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67cc06cb80 0x7f67cc0b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67d20bf640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67cc0d1f30 con 0x7f67cc0b6df0 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67d20bf640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f67cc0d2210 con 0x7f67cc06cb80 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67d20bf640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f67cc0d24f0 con 0x7f67cc06c7a0 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67c2ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67cc06cb80 0x7f67cc0b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67c2ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67cc06cb80 0x7f67cc0b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50944/0 (socket says 172.21.15.6:50944) 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67c2ffd640 1 -- 172.21.15.6:0/1522822527 learned_addr learned my addr 172.21.15.6:0/1522822527 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67cb7fe640 1 --2- 172.21.15.6:0/1522822527 >> v2:172.21.15.92:3300/0 conn(0x7f67cc06c7a0 0x7f67cc0d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67c2ffd640 1 -- 172.21.15.6:0/1522822527 >> v2:172.21.15.92:3300/0 conn(0x7f67cc06c7a0 msgr2=0x7f67cc0d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67c2ffd640 1 --2- 172.21.15.6:0/1522822527 >> v2:172.21.15.92:3300/0 conn(0x7f67cc06c7a0 0x7f67cc0d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67c2ffd640 1 -- 172.21.15.6:0/1522822527 >> v1:172.21.15.92:6789/0 conn(0x7f67cc0b6df0 legacy=0x7f67cc0b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.157+0000 7f67c2ffd640 1 -- 172.21.15.6:0/1522822527 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc0d27d0 con 0x7f67cc06cb80 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67c2ffd640 1 --2- 172.21.15.6:0/1522822527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06cb80 0x7f67cc0b95e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f67cc0d0fa0 tx=0x7f67b402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429c62e5c7d151c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67caffd640 1 -- 172.21.15.6:0/1522822527 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b402f850 con 0x7f67cc06cb80 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67caffd640 1 -- 172.21.15.6:0/1522822527 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b402f9f0 con 0x7f67cc06cb80 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67caffd640 1 -- 172.21.15.6:0/1522822527 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b402fdc0 con 0x7f67cc06cb80 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67d20bf640 1 -- 172.21.15.6:0/1522822527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06cb80 msgr2=0x7f67cc0b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1522822527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06cb80 0x7f67cc0b95e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f67cc0d0fa0 tx=0x7f67b402ba30 comp rx=0 tx=0).stop 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67d20bf640 1 -- 172.21.15.6:0/1522822527 shutdown_connections 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1522822527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06cb80 0x7f67cc0b95e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.158+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1522822527 >> v2:172.21.15.92:3300/0 conn(0x7f67cc06c7a0 0x7f67cc0d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.159+0000 7f67d20bf640 1 -- 172.21.15.6:0/1522822527 >> 172.21.15.6:0/1522822527 conn(0x7f67cc072ce0 msgr2=0x7f67cc0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.159+0000 7f67d20bf640 1 -- 172.21.15.6:0/1522822527 shutdown_connections 2024-03-18T14:42:50.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.159+0000 7f67d20bf640 1 -- 172.21.15.6:0/1522822527 wait complete. 2024-03-18T14:42:50.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.160+0000 7f67d20bf640 1 Processor -- start 2024-03-18T14:42:50.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.160+0000 7f67d20bf640 1 -- start start 2024-03-18T14:42:50.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67d20bf640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06c7a0 0x7f67cc1611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:50.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67d20bf640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67cc06cb80 0x7f67cc1f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67d20bf640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67cc0b6df0 0x7f67cc1f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67d20bf640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc0b9b20 con 0x7f67cc06c7a0 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06c7a0 0x7f67cc1611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67d20bf640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc06cfe0 con 0x7f67cc0b6df0 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06c7a0 0x7f67cc1611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50956/0 (socket says 172.21.15.6:50956) 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cb7fe640 1 -- 172.21.15.6:0/1690713688 learned_addr learned my addr 172.21.15.6:0/1690713688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67d20bf640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f67cc0b9ca0 con 0x7f67cc06cb80 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67c2ffd640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67cc06cb80 0x7f67cc1f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cbfff640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67cc0b6df0 0x7f67cc1f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cb7fe640 1 -- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67cc06cb80 msgr2=0x7f67cc1f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cb7fe640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67cc06cb80 0x7f67cc1f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cb7fe640 1 -- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67cc0b6df0 msgr2=0x7f67cc1f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.161+0000 7f67cb7fe640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67cc0b6df0 0x7f67cc1f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.162+0000 7f67cb7fe640 1 -- 172.21.15.6:0/1690713688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc0b43b0 con 0x7f67cc06c7a0 2024-03-18T14:42:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.162+0000 7f67cb7fe640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06c7a0 0x7f67cc1611c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00e9b0 tx=0x7f67bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.162+0000 7f67c8ff9640 1 -- 172.21.15.6:0/1690713688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc017070 con 0x7f67cc06c7a0 2024-03-18T14:42:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.162+0000 7f67c8ff9640 1 -- 172.21.15.6:0/1690713688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67bc00c9a0 con 0x7f67cc06c7a0 2024-03-18T14:42:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.162+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67cc15e330 con 0x7f67cc06c7a0 2024-03-18T14:42:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.162+0000 7f67c8ff9640 1 -- 172.21.15.6:0/1690713688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc00cd90 con 0x7f67cc06c7a0 2024-03-18T14:42:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.162+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67cc0c7f80 con 0x7f67cc06c7a0 2024-03-18T14:42:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.163+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67cc0cf2d0 con 0x7f67cc06c7a0 2024-03-18T14:42:50.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.163+0000 7f67c8ff9640 1 -- 172.21.15.6:0/1690713688 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f67bc01a070 con 0x7f67cc06c7a0 2024-03-18T14:42:50.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.164+0000 7f67c8ff9640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67ac06bbd0 0x7f67ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:50.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.164+0000 7f67c8ff9640 1 -- 172.21.15.6:0/1690713688 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f67bc08a7e0 con 0x7f67cc06c7a0 2024-03-18T14:42:50.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.164+0000 7f67c2ffd640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67ac06bbd0 0x7f67ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:50.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.164+0000 7f67c2ffd640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67ac06bbd0 0x7f67ac06e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f67b4036000 tx=0x7f67b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:50.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.168+0000 7f67c8ff9640 1 -- 172.21.15.6:0/1690713688 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f67cc0cf2d0 con 0x7f67cc06c7a0 2024-03-18T14:42:50.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.362+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f67cc15e850 con 0x7f67cc06c7a0 2024-03-18T14:42:50.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.983+0000 7f67c8ff9640 1 -- 172.21.15.6:0/1690713688 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f67bc058e80 con 0x7f67cc06c7a0 2024-03-18T14:42:50.985 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-18T14:42:50.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.986+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67ac06bbd0 msgr2=0x7f67ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:50.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.986+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67ac06bbd0 0x7f67ac06e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f67b4036000 tx=0x7f67b4008030 comp rx=0 tx=0).stop 2024-03-18T14:42:50.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.986+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06c7a0 msgr2=0x7f67cc1611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:50.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.986+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06c7a0 0x7f67cc1611c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00e9b0 tx=0x7f67bc00ee70 comp rx=0 tx=0).stop 2024-03-18T14:42:50.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 shutdown_connections 2024-03-18T14:42:50.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67ac06bbd0 0x7f67ac06e090 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67cc0b6df0 0x7f67cc1f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67cc06cb80 0x7f67cc1f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 --2- 172.21.15.6:0/1690713688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67cc06c7a0 0x7f67cc1611c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:50.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 >> 172.21.15.6:0/1690713688 conn(0x7f67cc072ce0 msgr2=0x7f67cc0b8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:50.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 shutdown_connections 2024-03-18T14:42:50.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:50.987+0000 7f67d20bf640 1 -- 172.21.15.6:0/1690713688 wait complete. 2024-03-18T14:42:51.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T14:42:51.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.109+0000 7f6cc2e9a640 1 Processor -- start 2024-03-18T14:42:51.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.109+0000 7f6cc2e9a640 1 -- start start 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc2e9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6cbc06cce0 0x7f6cbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc2e9a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc2e9a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cbc0cfd80 con 0x7f6cbc0ce3c0 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc2e9a640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6cbc0b2df0 con 0x7f6cbc06cce0 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc2e9a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc06d600 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc0c36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6cbc06cce0 0x7f6cbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc1437640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6cbc0ce3c0 0x7f6cbc0ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52756/0 (socket says 172.21.15.6:52756) 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc0c36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6cbc06cce0 0x7f6cbc06d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50972/0 (socket says 172.21.15.6:50972) 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc1437640 1 -- 172.21.15.6:0/3368330192 learned_addr learned my addr 172.21.15.6:0/3368330192 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cbbfff640 1 --2- 172.21.15.6:0/3368330192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc0c36640 1 -- 172.21.15.6:0/3368330192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0b17b0 msgr2=0x7f6cbc0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:51.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc0c36640 1 --2- 172.21.15.6:0/3368330192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc0c36640 1 -- 172.21.15.6:0/3368330192 >> v1:172.21.15.6:6789/0 conn(0x7f6cbc0ce3c0 legacy=0x7f6cbc0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cc0c36640 1 -- 172.21.15.6:0/3368330192 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc0b2270 con 0x7f6cbc06cce0 2024-03-18T14:42:51.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cbb7fe640 1 -- 172.21.15.6:0/3368330192 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 545770853 0 0) 0x7f6cbc0cfd80 con 0x7f6cbc0ce3c0 2024-03-18T14:42:51.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.110+0000 7f6cbbfff640 1 --2- 172.21.15.6:0/3368330192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.111+0000 7f6cc0c36640 1 --2- 172.21.15.6:0/3368330192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc06cce0 0x7f6cbc06d0c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00097a0 tx=0x7f6cb002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f1b21fd0053d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.111+0000 7f6cbb7fe640 1 -- 172.21.15.6:0/3368330192 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb00315a0 con 0x7f6cbc06cce0 2024-03-18T14:42:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.111+0000 7f6cbb7fe640 1 -- 172.21.15.6:0/3368330192 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb0037070 con 0x7f6cbc06cce0 2024-03-18T14:42:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.111+0000 7f6cbb7fe640 1 -- 172.21.15.6:0/3368330192 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0031960 con 0x7f6cbc06cce0 2024-03-18T14:42:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3368330192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc06cce0 msgr2=0x7f6cbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3368330192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc06cce0 0x7f6cbc06d0c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00097a0 tx=0x7f6cb002b840 comp rx=0 tx=0).stop 2024-03-18T14:42:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3368330192 shutdown_connections 2024-03-18T14:42:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3368330192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3368330192 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc06cce0 0x7f6cbc06d0c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3368330192 >> 172.21.15.6:0/3368330192 conn(0x7f6cbc08b130 msgr2=0x7f6cbc08b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3368330192 shutdown_connections 2024-03-18T14:42:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.112+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3368330192 wait complete. 2024-03-18T14:42:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.113+0000 7f6cc2e9a640 1 Processor -- start 2024-03-18T14:42:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.113+0000 7f6cc2e9a640 1 -- start start 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cc2e9a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cbc06cce0 0x7f6cbc0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cc2e9a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cc2e9a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0ce3c0 0x7f6cbc156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cc2e9a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc0b2b10 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cc2e9a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc0b2830 con 0x7f6cbc06cce0 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cc2e9a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc15e330 con 0x7f6cbc0ce3c0 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50976/0 (socket says 172.21.15.6:50976) 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 -- 172.21.15.6:0/3069537784 learned_addr learned my addr 172.21.15.6:0/3069537784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 -- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0ce3c0 msgr2=0x7f6cbc156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0ce3c0 0x7f6cbc156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 -- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cbc06cce0 msgr2=0x7f6cbc0b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cbc06cce0 0x7f6cbc0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 -- 172.21.15.6:0/3069537784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc0b57e0 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cbbfff640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b70b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800d800 tx=0x7f6ca800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.114+0000 7f6cb97fa640 1 -- 172.21.15.6:0/3069537784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8004030 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.115+0000 7f6cb97fa640 1 -- 172.21.15.6:0/3069537784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca80041d0 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.115+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc0b5ac0 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.115+0000 7f6cb97fa640 1 -- 172.21.15.6:0/3069537784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80096a0 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.115+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cbc0c69e0 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.116+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cbc09fa80 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.116+0000 7f6cb97fa640 1 -- 172.21.15.6:0/3069537784 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ca8009840 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.116+0000 7f6cb97fa640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ca006bbd0 0x7f6ca006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.117+0000 7f6cb97fa640 1 -- 172.21.15.6:0/3069537784 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f6ca808ac90 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.117+0000 7f6cc0c36640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ca006bbd0 0x7f6ca006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.117+0000 7f6cc0c36640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ca006bbd0 0x7f6ca006e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0036000 tx=0x7f6cb0030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.121+0000 7f6cb97fa640 1 -- 172.21.15.6:0/3069537784 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6ca80598a0 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.374+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f6cbc06c630 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.374+0000 7f6cb97fa640 1 -- 172.21.15.6:0/3069537784 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f6ca80591a0 con 0x7f6cbc0b17b0 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhltJMFFxAAsrfhgR5dgZaElpPSyFsH7Q==", 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.376 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlQARXIxAAcOlIn0OSzieJz4X0FMie1A==", 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhla1pOHhAAsDJ0P/0XWnoBakiPfEPYyA==", 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlADqzKhAAfG1fJKDAsis1WSLEAMQq0Q==", 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.378 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBpUvhlzhYhDxAATnOtwIggkZq8Cix5xsKd7A==", 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.379 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBsUvhlkHrEDRAAQHbA4maLjVfgH5XiBUVn0g==", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBvUvhlN5f9CxAAor/uSMxdAdq8E9k/Bcay0g==", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.380 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQByUvhls/oKChAANwFAbqxQRefnUxXp74nxaA==", 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQB4Uvhlj6qgEhAA5pcxWwon2a8wJcS4myHh/g==", 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.381 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQB7Uvhlx9p2ERAAP20961lxyrshPFvUDivuhA==", 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQB+Uvhl7I1SEBAAPkJjmK4erqFLRfqnbFkFqw==", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCBUvhldq3UDhAAgwbv2PzFpLHI4YE/Gpg/Eg==", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:42:51.383 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q==", 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhl1tD9NxAA7VTlgLnBw8A2KI2Rro0eAQ==", 2024-03-18T14:42:51.384 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlwRCfNBAAvhl6UTpZTW7GDuq1BP+hqQ==", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-18T14:42:51.385 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmUvhlByUyAhAANpMa65PITql7fLcswh+vog==", 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-18T14:42:51.386 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBkUvhlkIZbNRAAe4cTwo1Xf7UpmD6PIgTc0A==", 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-mds", 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCHUvhlznsECRAASBoFJATN2Yxj6VV9PvLBKg==", 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-mds" 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-mgr", 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCHUvhlOYQECRAAyUs2qsX6YfAVFmbk0+PWhw==", 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-osd", 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCHUvhl34oECRAA+CgxVV1zhzxCDDt3MyqWdw==", 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-osd" 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.388 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rbd", 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCHUvhlRZEECRAAFdG/Unrm4XREbvHLdNYGug==", 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCHUvhlC5gECRAAYr5bYyOJiKxaLqDnSOydsg==", 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rgw", 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCHUvhl1J4ECRAARuKlIFKmD0W1sn4I0y+OfA==", 2024-03-18T14:42:51.389 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlM+SZExAAawGogzVd1g2EvnpMmL0r1A==", 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:51.390 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlGVnoDhAAblQd6YniwXUXlv3N2eZvKQ==", 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-18T14:42:51.391 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:42:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ca006bbd0 msgr2=0x7f6ca006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ca006bbd0 0x7f6ca006e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0036000 tx=0x7f6cb0030000 comp rx=0 tx=0).stop 2024-03-18T14:42:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc0b17b0 msgr2=0x7f6cbc0b70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b70b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800d800 tx=0x7f6ca800dcc0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 shutdown_connections 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ca006bbd0 0x7f6ca006e090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cbc0ce3c0 0x7f6cbc156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cbc0b17b0 0x7f6cbc0b70b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 --2- 172.21.15.6:0/3069537784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cbc06cce0 0x7f6cbc0b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 >> 172.21.15.6:0/3069537784 conn(0x7f6cbc08b130 msgr2=0x7f6cbc08b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 shutdown_connections 2024-03-18T14:42:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.376+0000 7f6cc2e9a640 1 -- 172.21.15.6:0/3069537784 wait complete. 2024-03-18T14:42:51.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-18T14:42:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.494+0000 7f9e2c4ff640 1 Processor -- start 2024-03-18T14:42:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.494+0000 7f9e2c4ff640 1 -- start start 2024-03-18T14:42:51.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2c4ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e2415dd70 0x7f9e2415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2c4ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e240cecc0 0x7f9e240d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2c4ff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9e240d0910 0x7f9e240b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2c4ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9e240d12a0 con 0x7f9e240cecc0 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2c4ff640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9e24161a50 con 0x7f9e240d0910 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2c4ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9e24161d30 con 0x7f9e2415dd70 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e240cecc0 0x7f9e240d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e240cecc0 0x7f9e240d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50990/0 (socket says 172.21.15.6:50990) 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 -- 172.21.15.6:0/3845064364 learned_addr learned my addr 172.21.15.6:0/3845064364 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2a29b640 1 --2- 172.21.15.6:0/3845064364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e2415dd70 0x7f9e2415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2aa9c640 1 --2- 172.21.15.6:0/3845064364 >> v2:172.21.15.92:3300/0 conn(0x7f9e240d0910 0x7f9e240b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 -- 172.21.15.6:0/3845064364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e2415dd70 msgr2=0x7f9e2415ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 --2- 172.21.15.6:0/3845064364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e2415dd70 0x7f9e2415ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 -- 172.21.15.6:0/3845064364 >> v2:172.21.15.92:3300/0 conn(0x7f9e240d0910 msgr2=0x7f9e240b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:51.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 --2- 172.21.15.6:0/3845064364 >> v2:172.21.15.92:3300/0 conn(0x7f9e240d0910 0x7f9e240b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e29a9a640 1 -- 172.21.15.6:0/3845064364 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e24162010 con 0x7f9e240cecc0 2024-03-18T14:42:51.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.495+0000 7f9e2aa9c640 1 --2- 172.21.15.6:0/3845064364 >> v2:172.21.15.92:3300/0 conn(0x7f9e240d0910 0x7f9e240b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.496+0000 7f9e29a9a640 1 --2- 172.21.15.6:0/3845064364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240d03d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9e2408c550 tx=0x7f9e1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c90a86927ffbe7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.496+0000 7f9e29299640 1 -- 172.21.15.6:0/3845064364 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14035020 con 0x7f9e240cecc0 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.496+0000 7f9e29299640 1 -- 172.21.15.6:0/3845064364 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e14031870 con 0x7f9e240cecc0 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.496+0000 7f9e29299640 1 -- 172.21.15.6:0/3845064364 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14031c40 con 0x7f9e240cecc0 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.496+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/3845064364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 msgr2=0x7f9e240d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.496+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/3845064364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240d03d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9e2408c550 tx=0x7f9e1402b910 comp rx=0 tx=0).stop 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.497+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/3845064364 shutdown_connections 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.497+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/3845064364 >> v2:172.21.15.92:3300/0 conn(0x7f9e240d0910 0x7f9e240b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.497+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/3845064364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240d03d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.497+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/3845064364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e2415dd70 0x7f9e2415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.497+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/3845064364 >> 172.21.15.6:0/3845064364 conn(0x7f9e2406c900 msgr2=0x7f9e2406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.497+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/3845064364 shutdown_connections 2024-03-18T14:42:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.497+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/3845064364 wait complete. 2024-03-18T14:42:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 Processor -- start 2024-03-18T14:42:51.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 -- start start 2024-03-18T14:42:51.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e240d0910 0x7f9e240cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e2415dd70 0x7f9e24148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2415e150 con 0x7f9e240cecc0 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9e240d0da0 con 0x7f9e2415dd70 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.498+0000 7f9e2c4ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2415e2d0 con 0x7f9e240d0910 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51002/0 (socket says 172.21.15.6:51002) 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 -- 172.21.15.6:0/664006187 learned_addr learned my addr 172.21.15.6:0/664006187 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 -- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e240d0910 msgr2=0x7f9e240cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e240d0910 0x7f9e240cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 -- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e2415dd70 msgr2=0x7f9e24148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e2415dd70 0x7f9e24148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 -- 172.21.15.6:0/664006187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e240ccf50 con 0x7f9e240cecc0 2024-03-18T14:42:51.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2a29b640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240b3080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000d800 tx=0x7f9e2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e1affd640 1 -- 172.21.15.6:0/664006187 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000b6c0 con 0x7f9e240cecc0 2024-03-18T14:42:51.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e1affd640 1 -- 172.21.15.6:0/664006187 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e20019070 con 0x7f9e240cecc0 2024-03-18T14:42:51.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e240cd230 con 0x7f9e240cecc0 2024-03-18T14:42:51.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e1affd640 1 -- 172.21.15.6:0/664006187 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000ba80 con 0x7f9e240cecc0 2024-03-18T14:42:51.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.499+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e240b4ca0 con 0x7f9e240cecc0 2024-03-18T14:42:51.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.500+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e2404f5b0 con 0x7f9e240cecc0 2024-03-18T14:42:51.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.501+0000 7f9e1affd640 1 -- 172.21.15.6:0/664006187 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9e2000bc20 con 0x7f9e240cecc0 2024-03-18T14:42:51.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.501+0000 7f9e1affd640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9e0806bbd0 0x7f9e0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.501+0000 7f9e1affd640 1 -- 172.21.15.6:0/664006187 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f9e2008ad20 con 0x7f9e240cecc0 2024-03-18T14:42:51.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.501+0000 7f9e29a9a640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9e0806bbd0 0x7f9e0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.502+0000 7f9e29a9a640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9e0806bbd0 0x7f9e0806e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e14036000 tx=0x7f9e14030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.505+0000 7f9e1affd640 1 -- 172.21.15.6:0/664006187 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9e200599d0 con 0x7f9e240cecc0 2024-03-18T14:42:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.762+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f9e2406b8a0 con 0x7f9e240cecc0 2024-03-18T14:42:51.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.764+0000 7f9e1affd640 1 -- 172.21.15.6:0/664006187 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f9e200592d0 con 0x7f9e240cecc0 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9e0806bbd0 msgr2=0x7f9e0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9e0806bbd0 0x7f9e0806e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e14036000 tx=0x7f9e14030000 comp rx=0 tx=0).stop 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 msgr2=0x7f9e240b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240b3080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000d800 tx=0x7f9e2000dcc0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 shutdown_connections 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9e0806bbd0 0x7f9e0806e090 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e2415dd70 0x7f9e24148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e240d0910 0x7f9e240cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 --2- 172.21.15.6:0/664006187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e240cecc0 0x7f9e240b3080 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 >> 172.21.15.6:0/664006187 conn(0x7f9e2406c900 msgr2=0x7f9e24160af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:51.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 shutdown_connections 2024-03-18T14:42:51.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.766+0000 7f9e2c4ff640 1 -- 172.21.15.6:0/664006187 wait complete. 2024-03-18T14:42:51.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-18T14:42:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.879+0000 7f6acf675640 1 Processor -- start 2024-03-18T14:42:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.879+0000 7f6acf675640 1 -- start start 2024-03-18T14:42:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6acf675640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ac80b6df0 0x7f6ac80b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6acf675640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac80d3430 con 0x7f6ac806c7a0 2024-03-18T14:42:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6acf675640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac80d1f30 con 0x7f6ac80d2fd0 2024-03-18T14:42:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6acf675640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6ac80b9cf0 con 0x7f6ac80b6df0 2024-03-18T14:42:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6acd411640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6ac806c7a0 0x7f6ac806cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52768/0 (socket says 172.21.15.6:52768) 2024-03-18T14:42:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6acd411640 1 -- 172.21.15.6:0/989312218 learned_addr learned my addr 172.21.15.6:0/989312218 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762513612 0 0) 0x7f6ac80d3430 con 0x7f6ac806c7a0 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ab4003520 con 0x7f6ac806c7a0 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6acdc12640 1 --2- 172.21.15.6:0/989312218 >> v2:172.21.15.92:3300/0 conn(0x7f6ac80b6df0 0x7f6ac80b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2896965406 0 0) 0x7f6ab4003520 con 0x7f6ac806c7a0 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 >> v2:172.21.15.92:3300/0 conn(0x7f6ac80b6df0 msgr2=0x7f6ac80b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 --2- 172.21.15.6:0/989312218 >> v2:172.21.15.92:3300/0 conn(0x7f6ac80b6df0 0x7f6ac80b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 >> v1:172.21.15.92:6789/0 conn(0x7f6ac80d2fd0 legacy=0x7f6ac80b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac80d2210 con 0x7f6ac806c7a0 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f6ac4001670 con 0x7f6ac806c7a0 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.880+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 >> v1:172.21.15.6:6789/0 conn(0x7f6ac806c7a0 legacy=0x7f6ac806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6abffff640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ab40039b0 0x7f6ab4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6abffff640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ab4004800 0x7f6ab4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6abffff640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ab400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab400c040 con 0x7f6ab40051c0 2024-03-18T14:42:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab400c1f0 con 0x7f6ab40039b0 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab400c3f0 con 0x7f6ab4004800 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6acdc12640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ab400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6acd411640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ab40039b0 0x7f6ab4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6accc10640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ab4004800 0x7f6ab4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6acdc12640 1 -- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ab4004800 msgr2=0x7f6ab4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6acdc12640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ab4004800 0x7f6ab4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6acdc12640 1 -- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ab40039b0 msgr2=0x7f6ab4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6acdc12640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ab40039b0 0x7f6ab4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.881+0000 7f6acdc12640 1 -- 172.21.15.6:0/989312218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab4003520 con 0x7f6ab40051c0 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acdc12640 1 -- 172.21.15.6:0/989312218 reap_dead start 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acdc12640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ab400b9c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009fc0 tx=0x7f6ab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2b433765e89f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0009c10 con 0x7f6ab40051c0 2024-03-18T14:42:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac002cca0 con 0x7f6ab40051c0 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6abffff640 1 -- 172.21.15.6:0/989312218 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0003bd0 con 0x7f6ab40051c0 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acf675640 1 -- 172.21.15.6:0/989312218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 msgr2=0x7f6ab400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acf675640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ab400b9c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009fc0 tx=0x7f6ab4005830 comp rx=0 tx=0).stop 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acf675640 1 -- 172.21.15.6:0/989312218 shutdown_connections 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acf675640 1 --2- 172.21.15.6:0/989312218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ab400b9c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acf675640 1 -- 172.21.15.6:0/989312218 >> 172.21.15.6:0/989312218 conn(0x7f6ac8072ce0 msgr2=0x7f6ac80730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.882+0000 7f6acf675640 1 -- 172.21.15.6:0/989312218 shutdown_connections 2024-03-18T14:42:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.883+0000 7f6acf675640 1 -- 172.21.15.6:0/989312218 wait complete. 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 Processor -- start 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 -- start start 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ac8155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ac8160220 0x7f6ac80b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ac81606a0 0x7f6ac80b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac80d2ab0 con 0x7f6ab40051c0 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac80d27d0 con 0x7f6ac8160220 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acf675640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac80d37f0 con 0x7f6ac81606a0 2024-03-18T14:42:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.884+0000 7f6acd411640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ac8155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ac8155290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51022/0 (socket says 172.21.15.6:51022) 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 -- 172.21.15.6:0/3698315404 learned_addr learned my addr 172.21.15.6:0/3698315404 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acdc12640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ac81606a0 0x7f6ac80b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6accc10640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ac8160220 0x7f6ac80b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 -- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ac81606a0 msgr2=0x7f6ac80b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ac81606a0 0x7f6ac80b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 -- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ac8160220 msgr2=0x7f6ac80b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ac8160220 0x7f6ac80b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 -- 172.21.15.6:0/3698315404 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac815e230 con 0x7f6ab40051c0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acd411640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ac8155290 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4004f50 tx=0x7f6ac40074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6abdffb640 1 -- 172.21.15.6:0/3698315404 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4007740 con 0x7f6ab40051c0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6abdffb640 1 -- 172.21.15.6:0/3698315404 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac4018070 con 0x7f6ab40051c0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac815e510 con 0x7f6ab40051c0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6abdffb640 1 -- 172.21.15.6:0/3698315404 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400fb00 con 0x7f6ab40051c0 2024-03-18T14:42:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.885+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac815ead0 con 0x7f6ab40051c0 2024-03-18T14:42:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.886+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac804f5b0 con 0x7f6ab40051c0 2024-03-18T14:42:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.887+0000 7f6abdffb640 1 -- 172.21.15.6:0/3698315404 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ac400fca0 con 0x7f6ab40051c0 2024-03-18T14:42:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.887+0000 7f6abdffb640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6aac06bda0 0x7f6aac06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.887+0000 7f6abdffb640 1 -- 172.21.15.6:0/3698315404 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f6ac408a770 con 0x7f6ab40051c0 2024-03-18T14:42:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.887+0000 7f6accc10640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6aac06bda0 0x7f6aac06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.888+0000 7f6accc10640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6aac06bda0 0x7f6aac06e260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8000c00 tx=0x7f6ab8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:51.891+0000 7f6abdffb640 1 -- 172.21.15.6:0/3698315404 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6ac40594b0 con 0x7f6ab40051c0 2024-03-18T14:42:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.145+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f6ac815d680 con 0x7f6ab40051c0 2024-03-18T14:42:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.147+0000 7f6abdffb640 1 -- 172.21.15.6:0/3698315404 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f6ac405de20 con 0x7f6ab40051c0 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6aac06bda0 msgr2=0x7f6aac06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6aac06bda0 0x7f6aac06e260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8000c00 tx=0x7f6ab8009040 comp rx=0 tx=0).stop 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 msgr2=0x7f6ac8155290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ac8155290 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4004f50 tx=0x7f6ac40074f0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 shutdown_connections 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6aac06bda0 0x7f6aac06e260 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ac81606a0 0x7f6ac80b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ac8160220 0x7f6ac80b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 --2- 172.21.15.6:0/3698315404 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab40051c0 0x7f6ac8155290 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 >> 172.21.15.6:0/3698315404 conn(0x7f6ac8072ce0 msgr2=0x7f6ac80cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 shutdown_connections 2024-03-18T14:42:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.149+0000 7f6acf675640 1 -- 172.21.15.6:0/3698315404 wait complete. 2024-03-18T14:42:52.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-18T14:42:52.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.262+0000 7fe9090c0640 1 Processor -- start 2024-03-18T14:42:52.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.262+0000 7fe9090c0640 1 -- start start 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.262+0000 7fe9090c0640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe90415bcb0 0x7fe9040ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe9090c0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe9040d1bf0 0x7fe9040d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe9090c0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9040aeef0 con 0x7fe90415c090 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe9090c0640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe9040afb00 con 0x7fe90415bcb0 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe9090c0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe9040d2580 con 0x7fe9040d1bf0 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe902d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe90415bcb0 0x7fe9040ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe90259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe90415c090 0x7fe9040d04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52778/0 (socket says 172.21.15.6:52778) 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe90259c640 1 -- 172.21.15.6:0/3693329361 learned_addr learned my addr 172.21.15.6:0/3693329361 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe902d9d640 1 -- 172.21.15.6:0/3693329361 >> v2:172.21.15.92:3300/0 conn(0x7fe9040d1bf0 msgr2=0x7fe9040d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe902d9d640 1 --2- 172.21.15.6:0/3693329361 >> v2:172.21.15.92:3300/0 conn(0x7fe9040d1bf0 0x7fe9040d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe902d9d640 1 -- 172.21.15.6:0/3693329361 >> v1:172.21.15.6:6789/0 conn(0x7fe90415c090 legacy=0x7fe9040d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe901d9b640 1 -- 172.21.15.6:0/3693329361 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 454567658 0 0) 0x7fe9040aeef0 con 0x7fe90415c090 2024-03-18T14:42:52.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.263+0000 7fe902d9d640 1 -- 172.21.15.6:0/3693329361 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9040af1d0 con 0x7fe90415bcb0 2024-03-18T14:42:52.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe902d9d640 1 --2- 172.21.15.6:0/3693329361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040ae910 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe9040cc330 tx=0x7fe8f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c234166dca8bdbd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:52.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe901d9b640 1 -- 172.21.15.6:0/3693329361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40315a0 con 0x7fe90415bcb0 2024-03-18T14:42:52.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe901d9b640 1 -- 172.21.15.6:0/3693329361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8f4037070 con 0x7fe90415bcb0 2024-03-18T14:42:52.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe901d9b640 1 -- 172.21.15.6:0/3693329361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4031960 con 0x7fe90415bcb0 2024-03-18T14:42:52.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 -- 172.21.15.6:0/3693329361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 msgr2=0x7fe9040ae910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3693329361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040ae910 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe9040cc330 tx=0x7fe8f402b840 comp rx=0 tx=0).stop 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 -- 172.21.15.6:0/3693329361 shutdown_connections 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3693329361 >> v2:172.21.15.92:3300/0 conn(0x7fe9040d1bf0 0x7fe9040d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3693329361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040ae910 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 -- 172.21.15.6:0/3693329361 >> 172.21.15.6:0/3693329361 conn(0x7fe90408b320 msgr2=0x7fe90408b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 -- 172.21.15.6:0/3693329361 shutdown_connections 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.264+0000 7fe9090c0640 1 -- 172.21.15.6:0/3693329361 wait complete. 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.266+0000 7fe9090c0640 1 Processor -- start 2024-03-18T14:42:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.266+0000 7fe9090c0640 1 -- start start 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe9090c0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9040d1bf0 0x7fe9040cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe9090c0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe9090c0640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe90415c090 0x7fe9040cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe9090c0640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe9040aff90 con 0x7fe90415bcb0 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe9090c0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe9040af790 con 0x7fe9040d1bf0 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe9090c0640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe9040b0110 con 0x7fe90415c090 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51040/0 (socket says 172.21.15.6:51040) 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 -- 172.21.15.6:0/3272690392 learned_addr learned my addr 172.21.15.6:0/3272690392 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe902d9d640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9040d1bf0 0x7fe9040cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90359e640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe90415c090 0x7fe9040cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 -- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe90415c090 msgr2=0x7fe9040cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe90415c090 0x7fe9040cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 -- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9040d1bf0 msgr2=0x7fe9040cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9040d1bf0 0x7fe9040cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 -- 172.21.15.6:0/3272690392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe904156730 con 0x7fe90415bcb0 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe902d9d640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9040d1bf0 0x7fe9040cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90359e640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe90415c090 0x7fe9040cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe90259c640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040cb160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800c440 tx=0x7fe8f800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe8eb7fe640 1 -- 172.21.15.6:0/3272690392 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8018070 con 0x7fe90415bcb0 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe8eb7fe640 1 -- 172.21.15.6:0/3272690392 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8f80074e0 con 0x7fe90415bcb0 2024-03-18T14:42:52.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe8eb7fe640 1 -- 172.21.15.6:0/3272690392 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8004a20 con 0x7fe90415bcb0 2024-03-18T14:42:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.267+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe904156a10 con 0x7fe90415bcb0 2024-03-18T14:42:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.268+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9040c6790 con 0x7fe90415bcb0 2024-03-18T14:42:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.269+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90404f5b0 con 0x7fe90415bcb0 2024-03-18T14:42:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.269+0000 7fe8eb7fe640 1 -- 172.21.15.6:0/3272690392 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe8f801b070 con 0x7fe90415bcb0 2024-03-18T14:42:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.269+0000 7fe8eb7fe640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe8f006bbd0 0x7fe8f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.269+0000 7fe8eb7fe640 1 -- 172.21.15.6:0/3272690392 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fe8f8089b20 con 0x7fe90415bcb0 2024-03-18T14:42:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.270+0000 7fe902d9d640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe8f006bbd0 0x7fe8f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.270+0000 7fe902d9d640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe8f006bbd0 0x7fe8f006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4036000 tx=0x7fe8f4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:52.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.274+0000 7fe8eb7fe640 1 -- 172.21.15.6:0/3272690392 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe8f80587d0 con 0x7fe90415bcb0 2024-03-18T14:42:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.528+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7fe904159c70 con 0x7fe90415bcb0 2024-03-18T14:42:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.529+0000 7fe8eb7fe640 1 -- 172.21.15.6:0/3272690392 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7fe8f80580d0 con 0x7fe90415bcb0 2024-03-18T14:42:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe8f006bbd0 msgr2=0x7fe8f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe8f006bbd0 0x7fe8f006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4036000 tx=0x7fe8f4008030 comp rx=0 tx=0).stop 2024-03-18T14:42:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 msgr2=0x7fe9040cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040cb160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800c440 tx=0x7fe8f800c900 comp rx=0 tx=0).stop 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 shutdown_connections 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe8f006bbd0 0x7fe8f006e090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe90415c090 0x7fe9040cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe90415bcb0 0x7fe9040cb160 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 --2- 172.21.15.6:0/3272690392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9040d1bf0 0x7fe9040cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 >> 172.21.15.6:0/3272690392 conn(0x7fe90408b320 msgr2=0x7fe9040cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 shutdown_connections 2024-03-18T14:42:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.532+0000 7fe9090c0640 1 -- 172.21.15.6:0/3272690392 wait complete. 2024-03-18T14:42:52.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-18T14:42:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.645+0000 7f0a2359e640 1 Processor -- start 2024-03-18T14:42:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.645+0000 7f0a2359e640 1 -- start start 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2359e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a2415dd50 0x7f0a240b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2359e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0a240b5cd0 0x7f0a240ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2359e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a24161bc0 con 0x7f0a2415e130 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2359e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0a24161ea0 con 0x7f0a240b5cd0 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2359e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0a24162180 con 0x7f0a2415dd50 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a2415dd50 0x7f0a240b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a22d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0a240b5cd0 0x7f0a240ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a1bfff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0a2415e130 0x7f0a240b55c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40568/0 (socket says 172.21.15.6:40568) 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a1bfff640 1 -- 172.21.15.6:0/2084956407 learned_addr learned my addr 172.21.15.6:0/2084956407 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a21d9b640 1 -- 172.21.15.6:0/2084956407 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425359247 0 0) 0x7f0a24161bc0 con 0x7f0a2415e130 2024-03-18T14:42:52.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a21d9b640 1 -- 172.21.15.6:0/2084956407 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a10003520 con 0x7f0a2415e130 2024-03-18T14:42:52.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2259c640 1 -- 172.21.15.6:0/2084956407 >> v2:172.21.15.92:3300/0 conn(0x7f0a240b5cd0 msgr2=0x7f0a240ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:52.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2259c640 1 --2- 172.21.15.6:0/2084956407 >> v2:172.21.15.92:3300/0 conn(0x7f0a240b5cd0 0x7f0a240ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2259c640 1 -- 172.21.15.6:0/2084956407 >> v1:172.21.15.92:6789/0 conn(0x7f0a2415e130 legacy=0x7f0a240b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a2259c640 1 -- 172.21.15.6:0/2084956407 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a24162460 con 0x7f0a2415dd50 2024-03-18T14:42:52.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.646+0000 7f0a22d9d640 1 --2- 172.21.15.6:0/2084956407 >> v2:172.21.15.92:3300/0 conn(0x7f0a240b5cd0 0x7f0a240ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.647+0000 7f0a2259c640 1 --2- 172.21.15.6:0/2084956407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a2415dd50 0x7f0a240b2ff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0a140097a0 tx=0x7f0a1402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9672d3a3bf807f46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.647+0000 7f0a21d9b640 1 -- 172.21.15.6:0/2084956407 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a14035020 con 0x7f0a2415dd50 2024-03-18T14:42:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.647+0000 7f0a21d9b640 1 -- 172.21.15.6:0/2084956407 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a14031830 con 0x7f0a2415dd50 2024-03-18T14:42:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.647+0000 7f0a21d9b640 1 -- 172.21.15.6:0/2084956407 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a14031be0 con 0x7f0a2415dd50 2024-03-18T14:42:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 -- 172.21.15.6:0/2084956407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a2415dd50 msgr2=0x7f0a240b2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2084956407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a2415dd50 0x7f0a240b2ff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0a140097a0 tx=0x7f0a1402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 -- 172.21.15.6:0/2084956407 shutdown_connections 2024-03-18T14:42:52.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2084956407 >> v2:172.21.15.92:3300/0 conn(0x7f0a240b5cd0 0x7f0a240ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2084956407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a2415dd50 0x7f0a240b2ff0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 -- 172.21.15.6:0/2084956407 >> 172.21.15.6:0/2084956407 conn(0x7f0a240d05c0 msgr2=0x7f0a240d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:52.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 -- 172.21.15.6:0/2084956407 shutdown_connections 2024-03-18T14:42:52.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.648+0000 7f0a2359e640 1 -- 172.21.15.6:0/2084956407 wait complete. 2024-03-18T14:42:52.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.649+0000 7f0a2359e640 1 Processor -- start 2024-03-18T14:42:52.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.649+0000 7f0a2359e640 1 -- start start 2024-03-18T14:42:52.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a2359e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a240b5cd0 0x7f0a2415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a2359e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a2415dd50 0x7f0a2415fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a2359e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a24160400 0x7f0a24151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a2359e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0a240d1fd0 con 0x7f0a2415dd50 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a2359e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0a240d1e50 con 0x7f0a240b5cd0 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a2359e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0a240d2150 con 0x7f0a24160400 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a1bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a2415dd50 0x7f0a2415fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a1bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a2415dd50 0x7f0a2415fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51042/0 (socket says 172.21.15.6:51042) 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a1bfff640 1 -- 172.21.15.6:0/2496714294 learned_addr learned my addr 172.21.15.6:0/2496714294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a22d9d640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a24160400 0x7f0a24151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.650+0000 7f0a2259c640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a240b5cd0 0x7f0a2415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1bfff640 1 -- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a24160400 msgr2=0x7f0a24151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1bfff640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a24160400 0x7f0a24151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1bfff640 1 -- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a240b5cd0 msgr2=0x7f0a2415aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1bfff640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a240b5cd0 0x7f0a2415aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1bfff640 1 -- 172.21.15.6:0/2496714294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a240c96f0 con 0x7f0a2415dd50 2024-03-18T14:42:52.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a22d9d640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a24160400 0x7f0a24151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:52.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1bfff640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a2415dd50 0x7f0a2415fec0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00c390 tx=0x7f0a0c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:52.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1affd640 1 -- 172.21.15.6:0/2496714294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c017070 con 0x7f0a2415dd50 2024-03-18T14:42:52.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1affd640 1 -- 172.21.15.6:0/2496714294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0c005ce0 con 0x7f0a2415dd50 2024-03-18T14:42:52.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a240c99d0 con 0x7f0a2415dd50 2024-03-18T14:42:52.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a1affd640 1 -- 172.21.15.6:0/2496714294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0072f0 con 0x7f0a2415dd50 2024-03-18T14:42:52.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.651+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a240c9f90 con 0x7f0a2415dd50 2024-03-18T14:42:52.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.652+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2404f540 con 0x7f0a2415dd50 2024-03-18T14:42:52.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.652+0000 7f0a1affd640 1 -- 172.21.15.6:0/2496714294 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0a0c01a070 con 0x7f0a2415dd50 2024-03-18T14:42:52.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.653+0000 7f0a1affd640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a0406bbd0 0x7f0a0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:52.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.653+0000 7f0a1affd640 1 -- 172.21.15.6:0/2496714294 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f0a0c089d00 con 0x7f0a2415dd50 2024-03-18T14:42:52.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.653+0000 7f0a2259c640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a0406bbd0 0x7f0a0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:52.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.653+0000 7f0a2259c640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a0406bbd0 0x7f0a0406e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a14036000 tx=0x7f0a14008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:52.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.657+0000 7f0a1affd640 1 -- 172.21.15.6:0/2496714294 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0a0c0589b0 con 0x7f0a2415dd50 2024-03-18T14:42:52.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.907+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f0a240b5ab0 con 0x7f0a2415dd50 2024-03-18T14:42:52.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.909+0000 7f0a1affd640 1 -- 172.21.15.6:0/2496714294 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f0a0c0582b0 con 0x7f0a2415dd50 2024-03-18T14:42:52.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a0406bbd0 msgr2=0x7f0a0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a0406bbd0 0x7f0a0406e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a14036000 tx=0x7f0a14008030 comp rx=0 tx=0).stop 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a2415dd50 msgr2=0x7f0a2415fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a2415dd50 0x7f0a2415fec0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00c390 tx=0x7f0a0c00c850 comp rx=0 tx=0).stop 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 shutdown_connections 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0a0406bbd0 0x7f0a0406e090 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a24160400 0x7f0a24151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a2415dd50 0x7f0a2415fec0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 --2- 172.21.15.6:0/2496714294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a240b5cd0 0x7f0a2415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 >> 172.21.15.6:0/2496714294 conn(0x7f0a240d05c0 msgr2=0x7f0a240b7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 shutdown_connections 2024-03-18T14:42:52.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:52.912+0000 7f0a2359e640 1 -- 172.21.15.6:0/2496714294 wait complete. 2024-03-18T14:42:52.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-18T14:42:53.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 Processor -- start 2024-03-18T14:42:53.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 -- start start 2024-03-18T14:42:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd9d815bcb0 0x7fd9d80ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d815c090 0x7fd9d80d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd9d80d1bf0 0x7fd9d80d2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d80d25c0 con 0x7fd9d815bcb0 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d80afb00 con 0x7fd9d80d1bf0 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.026+0000 7fd9defc4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d80aeef0 con 0x7fd9d815c090 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd9d815bcb0 0x7fd9d80ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd9d815bcb0 0x7fd9d80ae910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51054/0 (socket says 172.21.15.6:51054) 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 -- 172.21.15.6:0/3123559503 learned_addr learned my addr 172.21.15.6:0/3123559503 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9d7fff640 1 --2- 172.21.15.6:0/3123559503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d815c090 0x7fd9d80d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dd561640 1 --2- 172.21.15.6:0/3123559503 >> v2:172.21.15.92:3300/0 conn(0x7fd9d80d1bf0 0x7fd9d80d2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 -- 172.21.15.6:0/3123559503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d815c090 msgr2=0x7fd9d80d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 --2- 172.21.15.6:0/3123559503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d815c090 0x7fd9d80d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 -- 172.21.15.6:0/3123559503 >> v2:172.21.15.92:3300/0 conn(0x7fd9d80d1bf0 msgr2=0x7fd9d80d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 --2- 172.21.15.6:0/3123559503 >> v2:172.21.15.92:3300/0 conn(0x7fd9d80d1bf0 0x7fd9d80d2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dcd60640 1 -- 172.21.15.6:0/3123559503 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d80af1d0 con 0x7fd9d815bcb0 2024-03-18T14:42:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9d7fff640 1 --2- 172.21.15.6:0/3123559503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d815c090 0x7fd9d80d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:53.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.027+0000 7fd9dd561640 1 --2- 172.21.15.6:0/3123559503 >> v2:172.21.15.92:3300/0 conn(0x7fd9d80d1bf0 0x7fd9d80d2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:53.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9dcd60640 1 --2- 172.21.15.6:0/3123559503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815bcb0 0x7fd9d80ae910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009870 tx=0x7fd9cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344d4e142d4d86ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9d77fe640 1 -- 172.21.15.6:0/3123559503 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc035020 con 0x7fd9d815bcb0 2024-03-18T14:42:53.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9d77fe640 1 -- 172.21.15.6:0/3123559503 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9cc031640 con 0x7fd9d815bcb0 2024-03-18T14:42:53.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9d77fe640 1 -- 172.21.15.6:0/3123559503 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc02d040 con 0x7fd9d815bcb0 2024-03-18T14:42:53.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9defc4640 1 -- 172.21.15.6:0/3123559503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815bcb0 msgr2=0x7fd9d80ae910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9defc4640 1 --2- 172.21.15.6:0/3123559503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815bcb0 0x7fd9d80ae910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009870 tx=0x7fd9cc02b8f0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9defc4640 1 -- 172.21.15.6:0/3123559503 shutdown_connections 2024-03-18T14:42:53.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9defc4640 1 --2- 172.21.15.6:0/3123559503 >> v2:172.21.15.92:3300/0 conn(0x7fd9d80d1bf0 0x7fd9d80d2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9defc4640 1 --2- 172.21.15.6:0/3123559503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d815c090 0x7fd9d80d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9defc4640 1 --2- 172.21.15.6:0/3123559503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815bcb0 0x7fd9d80ae910 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.028+0000 7fd9defc4640 1 -- 172.21.15.6:0/3123559503 >> 172.21.15.6:0/3123559503 conn(0x7fd9d808b320 msgr2=0x7fd9d808b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.029+0000 7fd9defc4640 1 -- 172.21.15.6:0/3123559503 shutdown_connections 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.029+0000 7fd9defc4640 1 -- 172.21.15.6:0/3123559503 wait complete. 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.030+0000 7fd9defc4640 1 Processor -- start 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.030+0000 7fd9defc4640 1 -- start start 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d80d1bf0 0x7fd9d81551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d815bcb0 0x7fd9d80cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815c090 0x7fd9d80cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d80d0a10 con 0x7fd9d815c090 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d80af790 con 0x7fd9d815bcb0 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d80cf670 con 0x7fd9d80d1bf0 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815c090 0x7fd9d80cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815c090 0x7fd9d80cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51060/0 (socket says 172.21.15.6:51060) 2024-03-18T14:42:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 -- 172.21.15.6:0/2663532182 learned_addr learned my addr 172.21.15.6:0/2663532182 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dcd60640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d80d1bf0 0x7fd9d81551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9d7fff640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d815bcb0 0x7fd9d80cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 -- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d80d1bf0 msgr2=0x7fd9d81551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d80d1bf0 0x7fd9d81551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 -- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d815bcb0 msgr2=0x7fd9d80cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d815bcb0 0x7fd9d80cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 -- 172.21.15.6:0/2663532182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d815d200 con 0x7fd9d815c090 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9dd561640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815c090 0x7fd9d80cf130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0007ef0 tx=0x7fd9d000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9d57fa640 1 -- 172.21.15.6:0/2663532182 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d001a070 con 0x7fd9d815c090 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9d57fa640 1 -- 172.21.15.6:0/2663532182 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d0003b90 con 0x7fd9d815c090 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d815d470 con 0x7fd9d815c090 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9d57fa640 1 -- 172.21.15.6:0/2663532182 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0004030 con 0x7fd9d815c090 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.031+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d815da30 con 0x7fd9d815c090 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.032+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d804f590 con 0x7fd9d815c090 2024-03-18T14:42:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.033+0000 7fd9d57fa640 1 -- 172.21.15.6:0/2663532182 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd9d001d070 con 0x7fd9d815c090 2024-03-18T14:42:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.033+0000 7fd9d57fa640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9bc06bbd0 0x7fd9bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.034+0000 7fd9d57fa640 1 -- 172.21.15.6:0/2663532182 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fd9d008ab30 con 0x7fd9d815c090 2024-03-18T14:42:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.034+0000 7fd9dcd60640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9bc06bbd0 0x7fd9bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.034+0000 7fd9dcd60640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9bc06bbd0 0x7fd9bc06e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc036000 tx=0x7fd9cc03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.038+0000 7fd9d57fa640 1 -- 172.21.15.6:0/2663532182 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd9d0010070 con 0x7fd9d815c090 2024-03-18T14:42:53.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.289+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fd9d80b7bb0 con 0x7fd9d815c090 2024-03-18T14:42:53.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.291+0000 7fd9d57fa640 1 -- 172.21.15.6:0/2663532182 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fd9d0059100 con 0x7fd9d815c090 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9bc06bbd0 msgr2=0x7fd9bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9bc06bbd0 0x7fd9bc06e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc036000 tx=0x7fd9cc03c040 comp rx=0 tx=0).stop 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815c090 msgr2=0x7fd9d80cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815c090 0x7fd9d80cf130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0007ef0 tx=0x7fd9d000d4a0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 shutdown_connections 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9bc06bbd0 0x7fd9bc06e090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9d815c090 0x7fd9d80cf130 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d815bcb0 0x7fd9d80cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 --2- 172.21.15.6:0/2663532182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9d80d1bf0 0x7fd9d81551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 >> 172.21.15.6:0/2663532182 conn(0x7fd9d808b320 msgr2=0x7fd9d80cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:53.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 shutdown_connections 2024-03-18T14:42:53.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.293+0000 7fd9defc4640 1 -- 172.21.15.6:0/2663532182 wait complete. 2024-03-18T14:42:53.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-18T14:42:53.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.406+0000 7f078df1b640 1 Processor -- start 2024-03-18T14:42:53.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.406+0000 7f078df1b640 1 -- start start 2024-03-18T14:42:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f078df1b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0788145610 0x7f07880cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f078df1b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07880cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f078df1b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07880cc390 con 0x7f07880cd690 2024-03-18T14:42:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f078df1b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f07881459f0 con 0x7f0788145610 2024-03-18T14:42:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f078df1b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f07880cc670 con 0x7f078815cf10 2024-03-18T14:42:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f0787fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f07880cd690 0x7f078806d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52782/0 (socket says 172.21.15.6:52782) 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f0787fff640 1 -- 172.21.15.6:0/588821002 learned_addr learned my addr 172.21.15.6:0/588821002 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f0786ffd640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07880cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f07877fe640 1 --2- 172.21.15.6:0/588821002 >> v2:172.21.15.92:3300/0 conn(0x7f0788145610 0x7f07880cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2214694111 0 0) 0x7f07880cc390 con 0x7f07880cd690 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.407+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0770003520 con 0x7f07880cd690 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3228248463 0 0) 0x7f0770003520 con 0x7f07880cd690 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 msgr2=0x7f07880cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07880cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 >> v2:172.21.15.92:3300/0 conn(0x7f0788145610 msgr2=0x7f07880cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 --2- 172.21.15.6:0/588821002 >> v2:172.21.15.92:3300/0 conn(0x7f0788145610 0x7f07880cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07880cc950 con 0x7f07880cd690 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f07800019f0 con 0x7f07880cd690 2024-03-18T14:42:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0786ffd640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07880cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 >> v1:172.21.15.6:6789/0 conn(0x7f07880cd690 legacy=0x7f078806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0770003ef0 0x7f0770004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0770004d40 0x7f07700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f07880cc670 con 0x7f0770003ef0 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f077000b9c0 con 0x7f0770004d40 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f077000bb40 con 0x7f078815cf10 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07877fe640 1 --2- 172.21.15.6:0/588821002 >> v2:172.21.15.92:3300/0 conn(0x7f0788145610 0x7f07880cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0786ffd640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0770003ef0 0x7f0770004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0787fff640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0770004d40 0x7f07700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07877fe640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0786ffd640 1 -- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 msgr2=0x7f07700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0786ffd640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07700039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0786ffd640 1 -- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0770004d40 msgr2=0x7f07700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0786ffd640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0770004d40 0x7f07700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f0786ffd640 1 -- 172.21.15.6:0/588821002 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0770003520 con 0x7f0770003ef0 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.408+0000 7f07877fe640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f078815cf10 0x7f07700039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.409+0000 7f0786ffd640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0770003ef0 0x7f0770004800 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f077c009c90 tx=0x7f0770005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e2e0962e825471 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.409+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c039070 con 0x7f0770003ef0 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.409+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077c02c8f0 con 0x7f0770003ef0 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.409+0000 7f07867fc640 1 -- 172.21.15.6:0/588821002 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c009860 con 0x7f0770003ef0 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.409+0000 7f078df1b640 1 -- 172.21.15.6:0/588821002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0770003ef0 msgr2=0x7f0770004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.409+0000 7f078df1b640 1 --2- 172.21.15.6:0/588821002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0770003ef0 0x7f0770004800 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f077c009c90 tx=0x7f0770005830 comp rx=0 tx=0).stop 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.410+0000 7f0787fff640 1 -- 172.21.15.6:0/588821002 reap_dead start 2024-03-18T14:42:53.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.410+0000 7f078df1b640 1 -- 172.21.15.6:0/588821002 shutdown_connections 2024-03-18T14:42:53.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.410+0000 7f078df1b640 1 -- 172.21.15.6:0/588821002 >> 172.21.15.6:0/588821002 conn(0x7f07880b1570 msgr2=0x7f07880b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:53.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.410+0000 7f078df1b640 1 -- 172.21.15.6:0/588821002 shutdown_connections 2024-03-18T14:42:53.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.410+0000 7f078df1b640 1 -- 172.21.15.6:0/588821002 wait complete. 2024-03-18T14:42:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.411+0000 7f078df1b640 1 Processor -- start 2024-03-18T14:42:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.411+0000 7f078df1b640 1 -- start start 2024-03-18T14:42:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0770003ef0 0x7f0788144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078813bd80 0x7f078813c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f078815b110 0x7f078815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f07880cf520 con 0x7f078813bd80 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f07880cf3a0 con 0x7f078815b110 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f078806e0d0 con 0x7f0770003ef0 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078813bd80 0x7f078813c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078813bd80 0x7f078813c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51078/0 (socket says 172.21.15.6:51078) 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 -- 172.21.15.6:0/792855148 learned_addr learned my addr 172.21.15.6:0/792855148 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f07877fe640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0770003ef0 0x7f0788144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0787fff640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f078815b110 0x7f078815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 -- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0770003ef0 msgr2=0x7f0788144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0770003ef0 0x7f0788144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 -- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f078815b110 msgr2=0x7f078815b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f078815b110 0x7f078815b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 -- 172.21.15.6:0/792855148 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078813c800 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f07877fe640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0770003ef0 0x7f0788144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f0786ffd640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078813bd80 0x7f078813c200 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f077c009c90 tx=0x7f077c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f076ffff640 1 -- 172.21.15.6:0/792855148 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c039050 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f076ffff640 1 -- 172.21.15.6:0/792855148 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077c045070 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f076ffff640 1 -- 172.21.15.6:0/792855148 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c009860 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078813cae0 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.412+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0788139b70 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.413+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078804f5b0 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.414+0000 7f076ffff640 1 -- 172.21.15.6:0/792855148 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f077c04a460 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.414+0000 7f076ffff640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f077406bc00 0x7f077406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.414+0000 7f076ffff640 1 -- 172.21.15.6:0/792855148 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f077c0aca90 con 0x7f078813bd80 2024-03-18T14:42:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.414+0000 7f07877fe640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f077406bc00 0x7f077406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.415+0000 7f07877fe640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f077406bc00 0x7f077406e0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0778006e80 tx=0x7f0778008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.419+0000 7f076ffff640 1 -- 172.21.15.6:0/792855148 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f077c07b6b0 con 0x7f078813bd80 2024-03-18T14:42:53.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.671+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f078806fa50 con 0x7f078813bd80 2024-03-18T14:42:53.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.673+0000 7f076ffff640 1 -- 172.21.15.6:0/792855148 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f077c07afb0 con 0x7f078813bd80 2024-03-18T14:42:53.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f077406bc00 msgr2=0x7f077406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f077406bc00 0x7f077406e0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0778006e80 tx=0x7f0778008040 comp rx=0 tx=0).stop 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078813bd80 msgr2=0x7f078813c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078813bd80 0x7f078813c200 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f077c009c90 tx=0x7f077c004730 comp rx=0 tx=0).stop 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 shutdown_connections 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f077406bc00 0x7f077406e0c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f078815b110 0x7f078815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078813bd80 0x7f078813c200 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 --2- 172.21.15.6:0/792855148 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0770003ef0 0x7f0788144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 >> 172.21.15.6:0/792855148 conn(0x7f07880b1570 msgr2=0x7f078815c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 shutdown_connections 2024-03-18T14:42:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.675+0000 7f078df1b640 1 -- 172.21.15.6:0/792855148 wait complete. 2024-03-18T14:42:53.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:53.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.787+0000 7f7017753640 1 Processor -- start 2024-03-18T14:42:53.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.787+0000 7f7017753640 1 -- start start 2024-03-18T14:42:53.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7017753640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 0x7f70100b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7017753640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f70100d0d90 0x7f70100b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7017753640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70100cef50 con 0x7f70100d3330 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7017753640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f70100b9b90 con 0x7f70100d0d90 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7017753640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f70100cf230 con 0x7f701015e170 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7014cee640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70100d3330 0x7f70100d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41522/0 (socket says 172.21.15.6:41522) 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7014cee640 1 -- 172.21.15.6:0/2997213862 learned_addr learned my addr 172.21.15.6:0/2997213862 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f70154ef640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 0x7f70100b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3044193925 0 0) 0x7f70100cef50 con 0x7f70100d3330 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ffc003520 con 0x7f70100d3330 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.788+0000 7f7015cf0640 1 --2- 172.21.15.6:0/2997213862 >> v2:172.21.15.92:3300/0 conn(0x7f70100d0d90 0x7f70100b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 400829508 0 0) 0x7f6ffc003520 con 0x7f70100d3330 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 msgr2=0x7f70100b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 0x7f70100b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 >> v2:172.21.15.92:3300/0 conn(0x7f70100d0d90 msgr2=0x7f70100b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 --2- 172.21.15.6:0/2997213862 >> v2:172.21.15.92:3300/0 conn(0x7f70100d0d90 0x7f70100b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70100d2b30 con 0x7f70100d3330 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f7000001670 con 0x7f70100d3330 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 >> v1:172.21.15.6:6789/0 conn(0x7f70100d3330 legacy=0x7f70100d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 0x7f6ffc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ffc003f30 0x7f6ffc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc004d40 0x7f6ffc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70100b9b90 con 0x7f6ffc003f30 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70100cf230 con 0x7f6ffc004d40 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6ffc00b9c0 con 0x7f701015e170 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7014cee640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ffc003f30 0x7f6ffc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f70154ef640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 0x7f6ffc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7015cf0640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc004d40 0x7f6ffc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7014cee640 1 -- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 msgr2=0x7f6ffc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7014cee640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f701015e170 0x7f6ffc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7014cee640 1 -- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc004d40 msgr2=0x7f6ffc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7014cee640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc004d40 0x7f6ffc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.789+0000 7f7014cee640 1 -- 172.21.15.6:0/2997213862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc003520 con 0x7f6ffc003f30 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.790+0000 7f7014cee640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ffc003f30 0x7f6ffc004800 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7000001050 tx=0x7f6ffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225c8637476d978 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.790+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000007d70 con 0x7f6ffc003f30 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.790+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7000030070 con 0x7f6ffc003f30 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.790+0000 7f7007fff640 1 -- 172.21.15.6:0/2997213862 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700002cd70 con 0x7f6ffc003f30 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.791+0000 7f7017753640 1 -- 172.21.15.6:0/2997213862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ffc003f30 msgr2=0x7f6ffc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.791+0000 7f7017753640 1 --2- 172.21.15.6:0/2997213862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ffc003f30 0x7f6ffc004800 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7000001050 tx=0x7f6ffc005830 comp rx=0 tx=0).stop 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.791+0000 7f7015cf0640 1 -- 172.21.15.6:0/2997213862 reap_dead start 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.791+0000 7f7017753640 1 -- 172.21.15.6:0/2997213862 shutdown_connections 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.791+0000 7f7017753640 1 -- 172.21.15.6:0/2997213862 >> 172.21.15.6:0/2997213862 conn(0x7f701006c900 msgr2=0x7f701006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.791+0000 7f7017753640 1 -- 172.21.15.6:0/2997213862 shutdown_connections 2024-03-18T14:42:53.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.791+0000 7f7017753640 1 -- 172.21.15.6:0/2997213862 wait complete. 2024-03-18T14:42:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.792+0000 7f7017753640 1 Processor -- start 2024-03-18T14:42:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.792+0000 7f7017753640 1 -- start start 2024-03-18T14:42:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7017753640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc003f30 0x7f701015b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7017753640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100b1d30 0x7f70100b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7017753640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f701015e7f0 0x7f701015ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7017753640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70100ceb70 con 0x7f70100b1d30 2024-03-18T14:42:53.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7017753640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70100d2820 con 0x7f701015e7f0 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7017753640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f70100cecf0 con 0x7f6ffc003f30 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100b1d30 0x7f70100b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100b1d30 0x7f70100b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57816/0 (socket says 172.21.15.6:57816) 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 -- 172.21.15.6:0/3424576297 learned_addr learned my addr 172.21.15.6:0/3424576297 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f70154ef640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc003f30 0x7f701015b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 -- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc003f30 msgr2=0x7f701015b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc003f30 0x7f701015b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 -- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f701015e7f0 msgr2=0x7f701015ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f701015e7f0 0x7f701015ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 -- 172.21.15.6:0/3424576297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701015f560 con 0x7f70100b1d30 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f70154ef640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc003f30 0x7f701015b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:53.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7014cee640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100b1d30 0x7f70100b21b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f700000c2d0 tx=0x7f7000000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7005ffb640 1 -- 172.21.15.6:0/3424576297 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000045070 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7005ffb640 1 -- 172.21.15.6:0/3424576297 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7000004bf0 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701015f840 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.793+0000 7f7005ffb640 1 -- 172.21.15.6:0/3424576297 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000007d50 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.794+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f701015fe00 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.795+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f701008afe0 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.795+0000 7f7005ffb640 1 -- 172.21.15.6:0/3424576297 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7000007070 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.795+0000 7f7005ffb640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ff406bbd0 0x7f6ff406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.796+0000 7f7005ffb640 1 -- 172.21.15.6:0/3424576297 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f70000ab8b0 con 0x7f70100b1d30 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.796+0000 7f70154ef640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ff406bbd0 0x7f6ff406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:53.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.796+0000 7f70154ef640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ff406bbd0 0x7f6ff406e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f700c0099e0 tx=0x7f700c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.800+0000 7f7005ffb640 1 -- 172.21.15.6:0/3424576297 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7000079ff0 con 0x7f70100b1d30 2024-03-18T14:42:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.982+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f701004f620 con 0x7f70100b1d30 2024-03-18T14:42:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.983+0000 7f7005ffb640 1 -- 172.21.15.6:0/3424576297 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13068 (secure 0 0 0) 0x7f700007e090 con 0x7f70100b1d30 2024-03-18T14:42:53.984 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:53.984 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:50.996575+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ff406bbd0 msgr2=0x7f6ff406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ff406bbd0 0x7f6ff406e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f700c0099e0 tx=0x7f700c008040 comp rx=0 tx=0).stop 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100b1d30 msgr2=0x7f70100b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100b1d30 0x7f70100b21b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f700000c2d0 tx=0x7f7000000fe0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 shutdown_connections 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6ff406bbd0 0x7f6ff406e090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f701015e7f0 0x7f701015ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70100b1d30 0x7f70100b21b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 --2- 172.21.15.6:0/3424576297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc003f30 0x7f701015b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 >> 172.21.15.6:0/3424576297 conn(0x7f701006c900 msgr2=0x7f70100b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 shutdown_connections 2024-03-18T14:42:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:53.985+0000 7f7017753640 1 -- 172.21.15.6:0/3424576297 wait complete. 2024-03-18T14:42:53.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:42:53.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:42:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.101+0000 7f386df90640 1 Processor -- start 2024-03-18T14:42:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.101+0000 7f386df90640 1 -- start start 2024-03-18T14:42:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f386df90640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f386815e150 0x7f38680d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f386df90640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f386806d400 0x7f3868099c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f386df90640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386809a180 0x7f38680cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f386df90640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f386815e530 con 0x7f386806d400 2024-03-18T14:42:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f386df90640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f386809a5b0 con 0x7f386815e150 2024-03-18T14:42:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f386df90640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f38680d0300 con 0x7f386809a180 2024-03-18T14:42:54.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f386806d400 0x7f3868099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f386806d400 0x7f3868099c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57832/0 (socket says 172.21.15.6:57832) 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 -- 172.21.15.6:0/857436896 learned_addr learned my addr 172.21.15.6:0/857436896 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3867fff640 1 --2- 172.21.15.6:0/857436896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386809a180 0x7f38680cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f38677fe640 1 --2- 172.21.15.6:0/857436896 >> v2:172.21.15.92:3300/0 conn(0x7f386815e150 0x7f38680d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 -- 172.21.15.6:0/857436896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386809a180 msgr2=0x7f38680cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 --2- 172.21.15.6:0/857436896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386809a180 0x7f38680cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 -- 172.21.15.6:0/857436896 >> v2:172.21.15.92:3300/0 conn(0x7f386815e150 msgr2=0x7f38680d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 --2- 172.21.15.6:0/857436896 >> v2:172.21.15.92:3300/0 conn(0x7f386815e150 0x7f38680d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.102+0000 7f3866ffd640 1 -- 172.21.15.6:0/857436896 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38680d05e0 con 0x7f386806d400 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.103+0000 7f38677fe640 1 --2- 172.21.15.6:0/857436896 >> v2:172.21.15.92:3300/0 conn(0x7f386815e150 0x7f38680d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:54.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.103+0000 7f3867fff640 1 --2- 172.21.15.6:0/857436896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386809a180 0x7f38680cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.103+0000 7f3866ffd640 1 --2- 172.21.15.6:0/857436896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386806d400 0x7f3868099c40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3868073000 tx=0x7f385c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdc6684562cd4a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.103+0000 7f38667fc640 1 -- 172.21.15.6:0/857436896 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c0317c0 con 0x7f386806d400 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.103+0000 7f38667fc640 1 -- 172.21.15.6:0/857436896 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f385c037070 con 0x7f386806d400 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.103+0000 7f38667fc640 1 -- 172.21.15.6:0/857436896 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c031b80 con 0x7f386806d400 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 -- 172.21.15.6:0/857436896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386806d400 msgr2=0x7f3868099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 --2- 172.21.15.6:0/857436896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386806d400 0x7f3868099c40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3868073000 tx=0x7f385c02b860 comp rx=0 tx=0).stop 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 -- 172.21.15.6:0/857436896 shutdown_connections 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 --2- 172.21.15.6:0/857436896 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386809a180 0x7f38680cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 --2- 172.21.15.6:0/857436896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386806d400 0x7f3868099c40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 --2- 172.21.15.6:0/857436896 >> v2:172.21.15.92:3300/0 conn(0x7f386815e150 0x7f38680d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 -- 172.21.15.6:0/857436896 >> 172.21.15.6:0/857436896 conn(0x7f38680b3650 msgr2=0x7f38680b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 -- 172.21.15.6:0/857436896 shutdown_connections 2024-03-18T14:42:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.104+0000 7f386df90640 1 -- 172.21.15.6:0/857436896 wait complete. 2024-03-18T14:42:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.105+0000 7f386df90640 1 Processor -- start 2024-03-18T14:42:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.105+0000 7f386df90640 1 -- start start 2024-03-18T14:42:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f386df90640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f386806d400 0x7f3868160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f386df90640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386809a180 0x7f3868157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f386df90640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386815e150 0x7f38680b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f386df90640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f38680d0e80 con 0x7f386809a180 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f386df90640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f38680d0ba0 con 0x7f386806d400 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f386df90640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f386806d890 con 0x7f386815e150 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386809a180 0x7f3868157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386809a180 0x7f3868157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57834/0 (socket says 172.21.15.6:57834) 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 -- 172.21.15.6:0/3811250779 learned_addr learned my addr 172.21.15.6:0/3811250779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 -- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386815e150 msgr2=0x7f38680b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386815e150 0x7f38680b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 -- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f386806d400 msgr2=0x7f3868160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f386806d400 0x7f3868160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.106+0000 7f3866ffd640 1 -- 172.21.15.6:0/3811250779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3868158150 con 0x7f386809a180 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.107+0000 7f3866ffd640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386809a180 0x7f3868157c10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f385c02b8a0 tx=0x7f385c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.107+0000 7f384ffff640 1 -- 172.21.15.6:0/3811250779 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c032650 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.107+0000 7f384ffff640 1 -- 172.21.15.6:0/3811250779 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f385c032810 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.107+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38680cb750 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.107+0000 7f384ffff640 1 -- 172.21.15.6:0/3811250779 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c032c00 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.107+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3868160730 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.108+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38680b60f0 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.108+0000 7f384ffff640 1 -- 172.21.15.6:0/3811250779 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f385c04c060 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.108+0000 7f384ffff640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f385406bbd0 0x7f385406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.109+0000 7f384ffff640 1 -- 172.21.15.6:0/3811250779 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f385c0ac3b0 con 0x7f386809a180 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.109+0000 7f38677fe640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f385406bbd0 0x7f385406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.109+0000 7f38677fe640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f385406bbd0 0x7f385406e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3858006fd0 tx=0x7f3858008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.113+0000 7f384ffff640 1 -- 172.21.15.6:0/3811250779 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f385c040070 con 0x7f386809a180 2024-03-18T14:42:54.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.296+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38680751b0 con 0x7f386809a180 2024-03-18T14:42:54.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.296+0000 7f384ffff640 1 -- 172.21.15.6:0/3811250779 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7f385c07b060 con 0x7f386809a180 2024-03-18T14:42:54.298 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f385406bbd0 msgr2=0x7f385406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f385406bbd0 0x7f385406e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3858006fd0 tx=0x7f3858008040 comp rx=0 tx=0).stop 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386809a180 msgr2=0x7f3868157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386809a180 0x7f3868157c10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f385c02b8a0 tx=0x7f385c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 shutdown_connections 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f385406bbd0 0x7f385406e090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f386815e150 0x7f38680b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f386809a180 0x7f3868157c10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 --2- 172.21.15.6:0/3811250779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f386806d400 0x7f3868160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 >> 172.21.15.6:0/3811250779 conn(0x7f38680b3650 msgr2=0x7f38680b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 shutdown_connections 2024-03-18T14:42:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.298+0000 7f386df90640 1 -- 172.21.15.6:0/3811250779 wait complete. 2024-03-18T14:42:54.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T14:42:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.410+0000 7f3443df4640 1 Processor -- start 2024-03-18T14:42:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.410+0000 7f3443df4640 1 -- start start 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3443df4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f343c15e020 0x7f343c162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3443df4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f343c0ced00 0x7f343c0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3443df4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c0d3210 0x7f343c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3443df4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f343c0b9660 con 0x7f343c15e020 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3443df4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f343c0ce390 con 0x7f343c0ced00 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3443df4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f343c0b9900 con 0x7f343c0d3210 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3441b90640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f343c15e020 0x7f343c162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3441b90640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f343c15e020 0x7f343c162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57838/0 (socket says 172.21.15.6:57838) 2024-03-18T14:42:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.411+0000 7f3441b90640 1 -- 172.21.15.6:0/162976203 learned_addr learned my addr 172.21.15.6:0/162976203 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.412+0000 7f3441b90640 1 -- 172.21.15.6:0/162976203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c0d3210 msgr2=0x7f343c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.412+0000 7f3441b90640 1 --2- 172.21.15.6:0/162976203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c0d3210 0x7f343c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.412+0000 7f3441b90640 1 -- 172.21.15.6:0/162976203 >> v2:172.21.15.92:3300/0 conn(0x7f343c0ced00 msgr2=0x7f343c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.412+0000 7f3441b90640 1 --2- 172.21.15.6:0/162976203 >> v2:172.21.15.92:3300/0 conn(0x7f343c0ced00 0x7f343c0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.412+0000 7f3441b90640 1 -- 172.21.15.6:0/162976203 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c0b9be0 con 0x7f343c15e020 2024-03-18T14:42:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.412+0000 7f3441b90640 1 --2- 172.21.15.6:0/162976203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c15e020 0x7f343c162380 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f343c0d4da0 tx=0x7f342c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee503c4bf246b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.412+0000 7f3440b8e640 1 -- 172.21.15.6:0/162976203 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c02c9e0 con 0x7f343c15e020 2024-03-18T14:42:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3440b8e640 1 -- 172.21.15.6:0/162976203 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f342c02cb80 con 0x7f343c15e020 2024-03-18T14:42:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3440b8e640 1 -- 172.21.15.6:0/162976203 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c0343f0 con 0x7f343c15e020 2024-03-18T14:42:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 -- 172.21.15.6:0/162976203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c15e020 msgr2=0x7f343c162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 --2- 172.21.15.6:0/162976203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c15e020 0x7f343c162380 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f343c0d4da0 tx=0x7f342c02be40 comp rx=0 tx=0).stop 2024-03-18T14:42:54.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 -- 172.21.15.6:0/162976203 shutdown_connections 2024-03-18T14:42:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 --2- 172.21.15.6:0/162976203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c0d3210 0x7f343c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 --2- 172.21.15.6:0/162976203 >> v2:172.21.15.92:3300/0 conn(0x7f343c0ced00 0x7f343c0d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 --2- 172.21.15.6:0/162976203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c15e020 0x7f343c162380 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 -- 172.21.15.6:0/162976203 >> 172.21.15.6:0/162976203 conn(0x7f343c072ce0 msgr2=0x7f343c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 -- 172.21.15.6:0/162976203 shutdown_connections 2024-03-18T14:42:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.413+0000 7f3443df4640 1 -- 172.21.15.6:0/162976203 wait complete. 2024-03-18T14:42:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.414+0000 7f3443df4640 1 Processor -- start 2024-03-18T14:42:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.414+0000 7f3443df4640 1 -- start start 2024-03-18T14:42:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3443df4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f343c0ced00 0x7f343c0b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3443df4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c0d3210 0x7f343c160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3443df4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c15e020 0x7f343c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3443df4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f343c0cf440 con 0x7f343c0d3210 2024-03-18T14:42:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3443df4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f343c0cf2c0 con 0x7f343c0ced00 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3443df4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f343c0d20a0 con 0x7f343c15e020 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c0d3210 0x7f343c160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c0d3210 0x7f343c160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57848/0 (socket says 172.21.15.6:57848) 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 -- 172.21.15.6:0/1764984434 learned_addr learned my addr 172.21.15.6:0/1764984434 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3441b90640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f343c0ced00 0x7f343c0b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f3442391640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c15e020 0x7f343c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 -- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c15e020 msgr2=0x7f343c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c15e020 0x7f343c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 -- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f343c0ced00 msgr2=0x7f343c0b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f343c0ced00 0x7f343c0b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.415+0000 7f344138f640 1 -- 172.21.15.6:0/1764984434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c0b37a0 con 0x7f343c0d3210 2024-03-18T14:42:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.416+0000 7f344138f640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c0d3210 0x7f343c160b20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f343000b410 tx=0x7f343000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.416+0000 7f343a7fc640 1 -- 172.21.15.6:0/1764984434 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430015020 con 0x7f343c0d3210 2024-03-18T14:42:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.416+0000 7f343a7fc640 1 -- 172.21.15.6:0/1764984434 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3430011850 con 0x7f343c0d3210 2024-03-18T14:42:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.416+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343c0b3a80 con 0x7f343c0d3210 2024-03-18T14:42:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.416+0000 7f343a7fc640 1 -- 172.21.15.6:0/1764984434 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430011c40 con 0x7f343c0d3210 2024-03-18T14:42:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.416+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343c0b4040 con 0x7f343c0d3210 2024-03-18T14:42:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.417+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f343c15caf0 con 0x7f343c0d3210 2024-03-18T14:42:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.417+0000 7f343a7fc640 1 -- 172.21.15.6:0/1764984434 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3430018020 con 0x7f343c0d3210 2024-03-18T14:42:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.418+0000 7f343a7fc640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f342806bbd0 0x7f342806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.418+0000 7f343a7fc640 1 -- 172.21.15.6:0/1764984434 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f343008a6f0 con 0x7f343c0d3210 2024-03-18T14:42:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.418+0000 7f3441b90640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f342806bbd0 0x7f342806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.418+0000 7f3441b90640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f342806bbd0 0x7f342806e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f342c035000 tx=0x7f342c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.422+0000 7f343a7fc640 1 -- 172.21.15.6:0/1764984434 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f343c15caf0 con 0x7f343c0d3210 2024-03-18T14:42:54.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.607+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f343c04f620 con 0x7f343c0d3210 2024-03-18T14:42:54.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.607+0000 7f343a7fc640 1 -- 172.21.15.6:0/1764984434 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7f343001f020 con 0x7f343c0d3210 2024-03-18T14:42:54.609 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:54.609 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-18T14:42:54.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:42:54.609 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-18T14:42:54.610 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-18T14:42:54.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:42:54.610 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f342806bbd0 msgr2=0x7f342806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f342806bbd0 0x7f342806e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f342c035000 tx=0x7f342c030000 comp rx=0 tx=0).stop 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c0d3210 msgr2=0x7f343c160b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c0d3210 0x7f343c160b20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f343000b410 tx=0x7f343000b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 shutdown_connections 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f342806bbd0 0x7f342806e090 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f343c15e020 0x7f343c151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f343c0d3210 0x7f343c160b20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 --2- 172.21.15.6:0/1764984434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f343c0ced00 0x7f343c0b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 >> 172.21.15.6:0/1764984434 conn(0x7f343c072ce0 msgr2=0x7f343c152b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:54.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 shutdown_connections 2024-03-18T14:42:54.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.609+0000 7f3443df4640 1 -- 172.21.15.6:0/1764984434 wait complete. 2024-03-18T14:42:54.621 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T14:42:54.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T14:42:54.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.725+0000 7f489c663640 1 Processor -- start 2024-03-18T14:42:54.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.725+0000 7f489c663640 1 -- start start 2024-03-18T14:42:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f489c663640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48940d1de0 0x7f48940d3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f489c663640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f489406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f489c663640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48940d22c0 con 0x7f489415e0b0 2024-03-18T14:42:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f489c663640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f489406d0e0 con 0x7f48940d1de0 2024-03-18T14:42:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f489c663640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f48940d25a0 con 0x7f489406c730 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f4899bfe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48940d1de0 0x7f48940d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f4899bfe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48940d1de0 0x7f48940d3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57852/0 (socket says 172.21.15.6:57852) 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.726+0000 7f4899bfe640 1 -- 172.21.15.6:0/3779140176 learned_addr learned my addr 172.21.15.6:0/3779140176 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.727+0000 7f489ac00640 1 --2- 172.21.15.6:0/3779140176 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f489406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.727+0000 7f4899bfe640 1 -- 172.21.15.6:0/3779140176 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 msgr2=0x7f489406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.727+0000 7f4899bfe640 1 --2- 172.21.15.6:0/3779140176 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f489406cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.727+0000 7f4899bfe640 1 -- 172.21.15.6:0/3779140176 >> v1:172.21.15.92:6789/0 conn(0x7f489415e0b0 legacy=0x7f48940b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.727+0000 7f4899bfe640 1 -- 172.21.15.6:0/3779140176 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48940d2880 con 0x7f48940d1de0 2024-03-18T14:42:54.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f4899bfe640 1 --2- 172.21.15.6:0/3779140176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48940d1de0 0x7f48940d3360 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4884009870 tx=0x7f488402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf7c994ebc94c80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f48993fd640 1 -- 172.21.15.6:0/3779140176 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488402f830 con 0x7f48940d1de0 2024-03-18T14:42:54.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f48993fd640 1 -- 172.21.15.6:0/3779140176 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488402f9d0 con 0x7f48940d1de0 2024-03-18T14:42:54.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f48993fd640 1 -- 172.21.15.6:0/3779140176 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488402fda0 con 0x7f48940d1de0 2024-03-18T14:42:54.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f489c663640 1 -- 172.21.15.6:0/3779140176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48940d1de0 msgr2=0x7f48940d3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f489c663640 1 --2- 172.21.15.6:0/3779140176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48940d1de0 0x7f48940d3360 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4884009870 tx=0x7f488402ba10 comp rx=0 tx=0).stop 2024-03-18T14:42:54.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f489c663640 1 -- 172.21.15.6:0/3779140176 shutdown_connections 2024-03-18T14:42:54.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f489c663640 1 --2- 172.21.15.6:0/3779140176 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f489406cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f489c663640 1 --2- 172.21.15.6:0/3779140176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48940d1de0 0x7f48940d3360 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.728+0000 7f489c663640 1 -- 172.21.15.6:0/3779140176 >> 172.21.15.6:0/3779140176 conn(0x7f4894072ce0 msgr2=0x7f48940730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:54.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.729+0000 7f489c663640 1 -- 172.21.15.6:0/3779140176 shutdown_connections 2024-03-18T14:42:54.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.729+0000 7f489c663640 1 -- 172.21.15.6:0/3779140176 wait complete. 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 Processor -- start 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 -- start start 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f48941e6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48940d1de0 0x7f4894160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f489415e0b0 0x7f48940b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f48940d38a0 con 0x7f489415e0b0 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f48940d2e40 con 0x7f48940d1de0 2024-03-18T14:42:54.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489c663640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f48941f7b10 con 0x7f489406c730 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.730+0000 7f489ac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f489415e0b0 0x7f48940b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f489415e0b0 0x7f48940b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57868/0 (socket says 172.21.15.6:57868) 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 -- 172.21.15.6:0/1708245799 learned_addr learned my addr 172.21.15.6:0/1708245799 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489a3ff640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f48941e6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f4899bfe640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48940d1de0 0x7f4894160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 -- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 msgr2=0x7f48941e6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f48941e6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 -- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48940d1de0 msgr2=0x7f4894160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48940d1de0 0x7f4894160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 -- 172.21.15.6:0/1708245799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48941f7df0 con 0x7f489415e0b0 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489ac00640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f489415e0b0 0x7f48940b58f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f488c007e60 tx=0x7f488c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f488affd640 1 -- 172.21.15.6:0/1708245799 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c014420 con 0x7f489415e0b0 2024-03-18T14:42:54.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f488affd640 1 -- 172.21.15.6:0/1708245799 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488c019070 con 0x7f489415e0b0 2024-03-18T14:42:54.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48941f80d0 con 0x7f489415e0b0 2024-03-18T14:42:54.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f488affd640 1 -- 172.21.15.6:0/1708245799 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c014820 con 0x7f489415e0b0 2024-03-18T14:42:54.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.731+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48941614b0 con 0x7f489415e0b0 2024-03-18T14:42:54.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.732+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f489408afe0 con 0x7f489415e0b0 2024-03-18T14:42:54.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.733+0000 7f488affd640 1 -- 172.21.15.6:0/1708245799 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f488c0149c0 con 0x7f489415e0b0 2024-03-18T14:42:54.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.733+0000 7f488affd640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:54.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.733+0000 7f488affd640 1 -- 172.21.15.6:0/1708245799 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f488c08b0f0 con 0x7f489415e0b0 2024-03-18T14:42:54.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.733+0000 7f489a3ff640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:54.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.734+0000 7f489a3ff640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4890001640 tx=0x7f4890009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:54.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.737+0000 7f488affd640 1 -- 172.21.15.6:0/1708245799 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f488c059da0 con 0x7f489415e0b0 2024-03-18T14:42:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:54.920+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f489404f620 con 0x7f489415e0b0 2024-03-18T14:42:55.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.294+0000 7f488affd640 1 -- 172.21.15.6:0/1708245799 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7f488c0596a0 con 0x7f489415e0b0 2024-03-18T14:42:55.296 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-18T14:42:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 msgr2=0x7f487806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4890001640 tx=0x7f4890009040 comp rx=0 tx=0).stop 2024-03-18T14:42:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f489415e0b0 msgr2=0x7f48940b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f489415e0b0 0x7f48940b58f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f488c007e60 tx=0x7f488c00e4b0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 shutdown_connections 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f487806bbd0 0x7f487806e090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f489415e0b0 0x7f48940b58f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48940d1de0 0x7f4894160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 --2- 172.21.15.6:0/1708245799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f489406c730 0x7f48941e6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 >> 172.21.15.6:0/1708245799 conn(0x7f4894072ce0 msgr2=0x7f48940730c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 shutdown_connections 2024-03-18T14:42:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.297+0000 7f489c663640 1 -- 172.21.15.6:0/1708245799 wait complete. 2024-03-18T14:42:55.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T14:42:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.423+0000 7f12f2044640 1 Processor -- start 2024-03-18T14:42:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.423+0000 7f12f2044640 1 -- start start 2024-03-18T14:42:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.423+0000 7f12f2044640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12ec09a350 0x7f12ec0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12f2044640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12ec0d0300 con 0x7f12ec0d13a0 2024-03-18T14:42:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12f2044640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12ec0d05e0 con 0x7f12ec15e150 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12f2044640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f12ec15e530 con 0x7f12ec09a350 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12ebfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12ec09a350 0x7f12ec0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12ebfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12ec09a350 0x7f12ec0cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57884/0 (socket says 172.21.15.6:57884) 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12eaffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12ec0d13a0 0x7f12ec099c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41538/0 (socket says 172.21.15.6:41538) 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12ebfff640 1 -- 172.21.15.6:0/1640223340 learned_addr learned my addr 172.21.15.6:0/1640223340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12ebfff640 1 -- 172.21.15.6:0/1640223340 >> v1:172.21.15.92:6789/0 conn(0x7f12ec15e150 legacy=0x7f12ec06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12ea7fc640 1 -- 172.21.15.6:0/1640223340 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 863465215 0 0) 0x7f12ec0d0300 con 0x7f12ec0d13a0 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12ebfff640 1 -- 172.21.15.6:0/1640223340 >> v1:172.21.15.6:6789/0 conn(0x7f12ec0d13a0 legacy=0x7f12ec099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.424+0000 7f12ebfff640 1 -- 172.21.15.6:0/1640223340 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec0d08c0 con 0x7f12ec09a350 2024-03-18T14:42:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.425+0000 7f12ebfff640 1 --2- 172.21.15.6:0/1640223340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec09a350 0x7f12ec0cc1b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009e50 tx=0x7f12e402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c95dfdeb4e7f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.425+0000 7f12ea7fc640 1 -- 172.21.15.6:0/1640223340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e402cb20 con 0x7f12ec09a350 2024-03-18T14:42:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.425+0000 7f12ea7fc640 1 -- 172.21.15.6:0/1640223340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e402ccc0 con 0x7f12ec09a350 2024-03-18T14:42:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.425+0000 7f12ea7fc640 1 -- 172.21.15.6:0/1640223340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4034610 con 0x7f12ec09a350 2024-03-18T14:42:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.425+0000 7f12f2044640 1 -- 172.21.15.6:0/1640223340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec09a350 msgr2=0x7f12ec0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:55.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.425+0000 7f12f2044640 1 --2- 172.21.15.6:0/1640223340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec09a350 0x7f12ec0cc1b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009e50 tx=0x7f12e402bf80 comp rx=0 tx=0).stop 2024-03-18T14:42:55.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.426+0000 7f12f2044640 1 -- 172.21.15.6:0/1640223340 shutdown_connections 2024-03-18T14:42:55.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.426+0000 7f12f2044640 1 --2- 172.21.15.6:0/1640223340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec09a350 0x7f12ec0cc1b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.426+0000 7f12f2044640 1 -- 172.21.15.6:0/1640223340 >> 172.21.15.6:0/1640223340 conn(0x7f12ec0b3650 msgr2=0x7f12ec0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:55.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.426+0000 7f12f2044640 1 -- 172.21.15.6:0/1640223340 shutdown_connections 2024-03-18T14:42:55.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.426+0000 7f12f2044640 1 -- 172.21.15.6:0/1640223340 wait complete. 2024-03-18T14:42:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.427+0000 7f12f2044640 1 Processor -- start 2024-03-18T14:42:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.427+0000 7f12f2044640 1 -- start start 2024-03-18T14:42:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.427+0000 7f12f2044640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12ec09a350 0x7f12ec0ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.427+0000 7f12f2044640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12ec0d13a0 0x7f12ec0c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12f2044640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec15e150 0x7f12ec0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12f2044640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec06c650 con 0x7f12ec15e150 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12f2044640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec0d0e80 con 0x7f12ec0d13a0 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12f2044640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f12ec06c7d0 con 0x7f12ec09a350 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec15e150 0x7f12ec0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec15e150 0x7f12ec0c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57892/0 (socket says 172.21.15.6:57892) 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 -- 172.21.15.6:0/1849822014 learned_addr learned my addr 172.21.15.6:0/1849822014 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12eaffd640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12ec0d13a0 0x7f12ec0c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12eb7fe640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12ec09a350 0x7f12ec0ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 -- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12ec09a350 msgr2=0x7f12ec0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12ec09a350 0x7f12ec0ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 -- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12ec0d13a0 msgr2=0x7f12ec0c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12ec0d13a0 0x7f12ec0c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 -- 172.21.15.6:0/1849822014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec1602c0 con 0x7f12ec15e150 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12ebfff640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec15e150 0x7f12ec0c78f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009e50 tx=0x7f12e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12d3fff640 1 -- 172.21.15.6:0/1849822014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4043030 con 0x7f12ec15e150 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12d3fff640 1 -- 172.21.15.6:0/1849822014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e4003990 con 0x7f12ec15e150 2024-03-18T14:42:55.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ec1605a0 con 0x7f12ec15e150 2024-03-18T14:42:55.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12d3fff640 1 -- 172.21.15.6:0/1849822014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4003b60 con 0x7f12ec15e150 2024-03-18T14:42:55.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.428+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ec160b60 con 0x7f12ec15e150 2024-03-18T14:42:55.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.429+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12ec0751b0 con 0x7f12ec15e150 2024-03-18T14:42:55.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.430+0000 7f12d3fff640 1 -- 172.21.15.6:0/1849822014 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f12e402cb30 con 0x7f12ec15e150 2024-03-18T14:42:55.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.430+0000 7f12d3fff640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d806bbd0 0x7f12d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:55.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.430+0000 7f12d3fff640 1 -- 172.21.15.6:0/1849822014 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7f12e40acec0 con 0x7f12ec15e150 2024-03-18T14:42:55.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.431+0000 7f12eb7fe640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d806bbd0 0x7f12d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:55.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.431+0000 7f12eb7fe640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d806bbd0 0x7f12d806e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f12dc000f60 tx=0x7f12dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:55.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.435+0000 7f12d3fff640 1 -- 172.21.15.6:0/1849822014 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f12e404d020 con 0x7f12ec15e150 2024-03-18T14:42:55.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:55.619+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f12ec09a130 con 0x7f12ec15e150 2024-03-18T14:42:56.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.300+0000 7f12d3fff640 1 -- 172.21.15.6:0/1849822014 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7f12e407b9f0 con 0x7f12ec15e150 2024-03-18T14:42:56.302 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-18T14:42:56.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.303+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d806bbd0 msgr2=0x7f12d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:56.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.303+0000 7f12f2044640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d806bbd0 0x7f12d806e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f12dc000f60 tx=0x7f12dc009040 comp rx=0 tx=0).stop 2024-03-18T14:42:56.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.303+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec15e150 msgr2=0x7f12ec0c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.303+0000 7f12f2044640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec15e150 0x7f12ec0c78f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009e50 tx=0x7f12e4004a40 comp rx=0 tx=0).stop 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 shutdown_connections 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f12d806bbd0 0x7f12d806e090 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12ec15e150 0x7f12ec0c78f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12ec0d13a0 0x7f12ec0c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 --2- 172.21.15.6:0/1849822014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12ec09a350 0x7f12ec0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 >> 172.21.15.6:0/1849822014 conn(0x7f12ec0b3650 msgr2=0x7f12ec0b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 shutdown_connections 2024-03-18T14:42:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.304+0000 7f12f2044640 1 -- 172.21.15.6:0/1849822014 wait complete. 2024-03-18T14:42:56.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T14:42:56.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.438+0000 7f719e86e640 1 Processor -- start 2024-03-18T14:42:56.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.438+0000 7f719e86e640 1 -- start start 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.439+0000 7f719e86e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f719815ead0 0x7f7198166920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.439+0000 7f719e86e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71980afef0 con 0x7f719815cf10 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.439+0000 7f719e86e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71980b01d0 con 0x7f7198071ad0 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.439+0000 7f719e86e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f719815d370 con 0x7f719815ead0 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.439+0000 7f719ce0b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f719815ead0 0x7f7198166920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.439+0000 7f719ce0b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f719815ead0 0x7f7198166920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57894/0 (socket says 172.21.15.6:57894) 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.440+0000 7f719ce0b640 1 -- 172.21.15.6:0/3973004474 learned_addr learned my addr 172.21.15.6:0/3973004474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.440+0000 7f719ce0b640 1 -- 172.21.15.6:0/3973004474 >> v1:172.21.15.92:6789/0 conn(0x7f7198071ad0 legacy=0x7f7198071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:56.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.440+0000 7f719ce0b640 1 -- 172.21.15.6:0/3973004474 >> v1:172.21.15.6:6789/0 conn(0x7f719815cf10 legacy=0x7f71980af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:56.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.440+0000 7f719ce0b640 1 -- 172.21.15.6:0/3973004474 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71980b04b0 con 0x7f719815ead0 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719ce0b640 1 --2- 172.21.15.6:0/3973004474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815ead0 0x7f7198166920 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7190009e50 tx=0x7f719002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c9ad054dd74e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f7196ffd640 1 -- 172.21.15.6:0/3973004474 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719002cbb0 con 0x7f719815ead0 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f7196ffd640 1 -- 172.21.15.6:0/3973004474 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719002cd50 con 0x7f719815ead0 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f7196ffd640 1 -- 172.21.15.6:0/3973004474 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7190034610 con 0x7f719815ead0 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719e86e640 1 -- 172.21.15.6:0/3973004474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815ead0 msgr2=0x7f7198166920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719e86e640 1 --2- 172.21.15.6:0/3973004474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815ead0 0x7f7198166920 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7190009e50 tx=0x7f719002c010 comp rx=0 tx=0).stop 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719e86e640 1 -- 172.21.15.6:0/3973004474 shutdown_connections 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719e86e640 1 --2- 172.21.15.6:0/3973004474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815ead0 0x7f7198166920 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719e86e640 1 -- 172.21.15.6:0/3973004474 >> 172.21.15.6:0/3973004474 conn(0x7f719806b1c0 msgr2=0x7f719806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719e86e640 1 -- 172.21.15.6:0/3973004474 shutdown_connections 2024-03-18T14:42:56.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.441+0000 7f719e86e640 1 -- 172.21.15.6:0/3973004474 wait complete. 2024-03-18T14:42:56.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.442+0000 7f719e86e640 1 Processor -- start 2024-03-18T14:42:56.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f719e86e640 1 -- start start 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f719e86e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7198071ad0 0x7f7198142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f719e86e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815cf10 0x7f719813ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f719e86e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f719815ead0 0x7f719813cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f719e86e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7198159a60 con 0x7f719815cf10 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f719e86e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f71980afc90 con 0x7f719815ead0 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f719e86e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7198159be0 con 0x7f7198071ad0 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f71977fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815cf10 0x7f719813ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:56.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f71977fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815cf10 0x7f719813ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57902/0 (socket says 172.21.15.6:57902) 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.443+0000 7f71977fe640 1 -- 172.21.15.6:0/842377409 learned_addr learned my addr 172.21.15.6:0/842377409 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f719ce0b640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f719815ead0 0x7f719813cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f7197fff640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7198071ad0 0x7f7198142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f71977fe640 1 -- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7198071ad0 msgr2=0x7f7198142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f71977fe640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7198071ad0 0x7f7198142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f71977fe640 1 -- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f719815ead0 msgr2=0x7f719813cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f71977fe640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f719815ead0 0x7f719813cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f71977fe640 1 -- 172.21.15.6:0/842377409 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71980add40 con 0x7f719815cf10 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f7197fff640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7198071ad0 0x7f7198142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f71977fe640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815cf10 0x7f719813ca80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c450 tx=0x7f718c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f7194ff9640 1 -- 172.21.15.6:0/842377409 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c0052d0 con 0x7f719815cf10 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f7194ff9640 1 -- 172.21.15.6:0/842377409 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f718c005470 con 0x7f719815cf10 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71980ae020 con 0x7f719815cf10 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f7194ff9640 1 -- 172.21.15.6:0/842377409 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c012870 con 0x7f719815cf10 2024-03-18T14:42:56.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.444+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71980ae5e0 con 0x7f719815cf10 2024-03-18T14:42:56.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.445+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71980ae820 con 0x7f719815cf10 2024-03-18T14:42:56.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.446+0000 7f7194ff9640 1 -- 172.21.15.6:0/842377409 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f718c012a10 con 0x7f719815cf10 2024-03-18T14:42:56.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.446+0000 7f7194ff9640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f718406bbd0 0x7f718406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:56.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.446+0000 7f7194ff9640 1 -- 172.21.15.6:0/842377409 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7f718c08a720 con 0x7f719815cf10 2024-03-18T14:42:56.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.446+0000 7f7197fff640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f718406bbd0 0x7f718406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:56.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.447+0000 7f7197fff640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f718406bbd0 0x7f718406e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7188001700 tx=0x7f7188009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:56.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.451+0000 7f7194ff9640 1 -- 172.21.15.6:0/842377409 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f71980ae820 con 0x7f719815cf10 2024-03-18T14:42:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:56.671+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7198068c60 con 0x7f719815cf10 2024-03-18T14:42:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.302+0000 7f7194ff9640 1 -- 172.21.15.6:0/842377409 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) ==== 172+0+0 (secure 0 0 0) 0x7f7198068c60 con 0x7f719815cf10 2024-03-18T14:42:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 5 and data pool 6 2024-03-18T14:42:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.306+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f718406bbd0 msgr2=0x7f718406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.306+0000 7f719e86e640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f718406bbd0 0x7f718406e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7188001700 tx=0x7f7188009040 comp rx=0 tx=0).stop 2024-03-18T14:42:57.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.306+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815cf10 msgr2=0x7f719813ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.306+0000 7f719e86e640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815cf10 0x7f719813ca80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c450 tx=0x7f718c00c910 comp rx=0 tx=0).stop 2024-03-18T14:42:57.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.307+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 shutdown_connections 2024-03-18T14:42:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.307+0000 7f719e86e640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f718406bbd0 0x7f718406e090 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.307+0000 7f719e86e640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f719815ead0 0x7f719813cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.307+0000 7f719e86e640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f719815cf10 0x7f719813ca80 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.307+0000 7f719e86e640 1 --2- 172.21.15.6:0/842377409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7198071ad0 0x7f7198142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.307+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 >> 172.21.15.6:0/842377409 conn(0x7f719806b1c0 msgr2=0x7f71980ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:57.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.309+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 shutdown_connections 2024-03-18T14:42:57.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.309+0000 7f719e86e640 1 -- 172.21.15.6:0/842377409 wait complete. 2024-03-18T14:42:57.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.449+0000 7fd6b6937640 1 Processor -- start 2024-03-18T14:42:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.449+0000 7fd6b6937640 1 -- start start 2024-03-18T14:42:57.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.449+0000 7fd6b6937640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b015ab10 0x7fd6b00b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.449+0000 7fd6b6937640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd6b00ae840 0x7fd6b00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.449+0000 7fd6b6937640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6b006f120 con 0x7fd6b00cb680 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.449+0000 7fd6b6937640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd6b006fca0 con 0x7fd6b00ae840 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.449+0000 7fd6b6937640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b006f400 con 0x7fd6b015ab10 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6b4ed4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd6b00cb680 0x7fd6b00cc260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41558/0 (socket says 172.21.15.6:41558) 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6b4ed4640 1 -- 172.21.15.6:0/3893086629 learned_addr learned my addr 172.21.15.6:0/3893086629 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6affff640 1 --2- 172.21.15.6:0/3893086629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b015ab10 0x7fd6b00b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6af7fe640 1 --2- 172.21.15.6:0/3893086629 >> v2:172.21.15.92:3300/0 conn(0x7fd6b00ae840 0x7fd6b00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6aeffd640 1 -- 172.21.15.6:0/3893086629 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883962551 0 0) 0x7fd6b006f120 con 0x7fd6b00cb680 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6aeffd640 1 -- 172.21.15.6:0/3893086629 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd698003520 con 0x7fd6b00cb680 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6affff640 1 -- 172.21.15.6:0/3893086629 >> v2:172.21.15.92:3300/0 conn(0x7fd6b00ae840 msgr2=0x7fd6b00cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6affff640 1 --2- 172.21.15.6:0/3893086629 >> v2:172.21.15.92:3300/0 conn(0x7fd6b00ae840 0x7fd6b00cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6affff640 1 -- 172.21.15.6:0/3893086629 >> v1:172.21.15.6:6789/0 conn(0x7fd6b00cb680 legacy=0x7fd6b00cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6affff640 1 -- 172.21.15.6:0/3893086629 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b006f6e0 con 0x7fd6b015ab10 2024-03-18T14:42:57.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.450+0000 7fd6af7fe640 1 --2- 172.21.15.6:0/3893086629 >> v2:172.21.15.92:3300/0 conn(0x7fd6b00ae840 0x7fd6b00cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.451+0000 7fd6affff640 1 --2- 172.21.15.6:0/3893086629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b015ab10 0x7fd6b00b51b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd6b006fe20 tx=0x7fd6a002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7817b64d59261e55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.451+0000 7fd6aeffd640 1 -- 172.21.15.6:0/3893086629 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0035020 con 0x7fd6b015ab10 2024-03-18T14:42:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.451+0000 7fd6aeffd640 1 -- 172.21.15.6:0/3893086629 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a0031800 con 0x7fd6b015ab10 2024-03-18T14:42:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.451+0000 7fd6aeffd640 1 -- 172.21.15.6:0/3893086629 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0031bd0 con 0x7fd6b015ab10 2024-03-18T14:42:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.451+0000 7fd6b6937640 1 -- 172.21.15.6:0/3893086629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b015ab10 msgr2=0x7fd6b00b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.451+0000 7fd6b6937640 1 --2- 172.21.15.6:0/3893086629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b015ab10 0x7fd6b00b51b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd6b006fe20 tx=0x7fd6a002b8a0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.452+0000 7fd6b6937640 1 -- 172.21.15.6:0/3893086629 shutdown_connections 2024-03-18T14:42:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.452+0000 7fd6b6937640 1 --2- 172.21.15.6:0/3893086629 >> v2:172.21.15.92:3300/0 conn(0x7fd6b00ae840 0x7fd6b00cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.452+0000 7fd6b6937640 1 --2- 172.21.15.6:0/3893086629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b015ab10 0x7fd6b00b51b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.452+0000 7fd6b6937640 1 -- 172.21.15.6:0/3893086629 >> 172.21.15.6:0/3893086629 conn(0x7fd6b00b1470 msgr2=0x7fd6b00b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.452+0000 7fd6b6937640 1 -- 172.21.15.6:0/3893086629 shutdown_connections 2024-03-18T14:42:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.452+0000 7fd6b6937640 1 -- 172.21.15.6:0/3893086629 wait complete. 2024-03-18T14:42:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.453+0000 7fd6b6937640 1 Processor -- start 2024-03-18T14:42:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.453+0000 7fd6b6937640 1 -- start start 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00ae840 0x7fd6b01f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b00cb680 0x7fd6b015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b015ab10 0x7fd6b00a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b015aef0 con 0x7fd6b015ab10 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b00b56f0 con 0x7fd6b00ae840 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b015ca00 con 0x7fd6b00cb680 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b015ab10 0x7fd6b00a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b015ab10 0x7fd6b00a94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57912/0 (socket says 172.21.15.6:57912) 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 -- 172.21.15.6:0/2897993819 learned_addr learned my addr 172.21.15.6:0/2897993819 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6af7fe640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b00cb680 0x7fd6b015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6affff640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00ae840 0x7fd6b01f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 -- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b00cb680 msgr2=0x7fd6b015c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b00cb680 0x7fd6b015c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 -- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00ae840 msgr2=0x7fd6b01f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00ae840 0x7fd6b01f3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 -- 172.21.15.6:0/2897993819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b01f41f0 con 0x7fd6b015ab10 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b4ed4640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b015ab10 0x7fd6b00a94b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800cb40 tx=0x7fd6a800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6acff9640 1 -- 172.21.15.6:0/2897993819 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8017070 con 0x7fd6b015ab10 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6acff9640 1 -- 172.21.15.6:0/2897993819 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a800aca0 con 0x7fd6b015ab10 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b01f44d0 con 0x7fd6b015ab10 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6acff9640 1 -- 172.21.15.6:0/2897993819 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8012620 con 0x7fd6b015ab10 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.454+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b01f0750 con 0x7fd6b015ab10 2024-03-18T14:42:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.455+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b0073310 con 0x7fd6b015ab10 2024-03-18T14:42:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.456+0000 7fd6acff9640 1 -- 172.21.15.6:0/2897993819 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd6a801a070 con 0x7fd6b015ab10 2024-03-18T14:42:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.456+0000 7fd6acff9640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd69c06bc00 0x7fd69c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.456+0000 7fd6acff9640 1 -- 172.21.15.6:0/2897993819 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd6a808a9a0 con 0x7fd6b015ab10 2024-03-18T14:42:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.456+0000 7fd6affff640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd69c06bc00 0x7fd69c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.457+0000 7fd6affff640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd69c06bc00 0x7fd69c06e0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0036000 tx=0x7fd6a003c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.461+0000 7fd6acff9640 1 -- 172.21.15.6:0/2897993819 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd6a8058cc0 con 0x7fd6b015ab10 2024-03-18T14:42:57.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.643+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6b004f620 con 0x7fd6b015ab10 2024-03-18T14:42:57.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.644+0000 7fd6acff9640 1 -- 172.21.15.6:0/2897993819 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16859 (secure 0 0 0) 0x7fd6a805cd60 con 0x7fd6b015ab10 2024-03-18T14:42:57.646 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:57.646 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:56.672498+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd69c06bc00 msgr2=0x7fd69c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd69c06bc00 0x7fd69c06e0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0036000 tx=0x7fd6a003c040 comp rx=0 tx=0).stop 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b015ab10 msgr2=0x7fd6b00a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b015ab10 0x7fd6b00a94b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800cb40 tx=0x7fd6a800a060 comp rx=0 tx=0).stop 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 shutdown_connections 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd69c06bc00 0x7fd69c06e0c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6b015ab10 0x7fd6b00a94b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6b00cb680 0x7fd6b015c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 --2- 172.21.15.6:0/2897993819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6b00ae840 0x7fd6b01f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 >> 172.21.15.6:0/2897993819 conn(0x7fd6b00b1470 msgr2=0x7fd6b00c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 shutdown_connections 2024-03-18T14:42:57.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.646+0000 7fd6b6937640 1 -- 172.21.15.6:0/2897993819 wait complete. 2024-03-18T14:42:57.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:57.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.758+0000 7f7d3c1b9640 1 Processor -- start 2024-03-18T14:42:57.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.758+0000 7f7d3c1b9640 1 -- start start 2024-03-18T14:42:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d3c1b9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d340ce040 0x7f7d340cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d3c1b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d340cc870 con 0x7f7d3415bd40 2024-03-18T14:42:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d3c1b9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d340d0500 con 0x7f7d340d1f60 2024-03-18T14:42:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d3c1b9640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7d340d07e0 con 0x7f7d340ce040 2024-03-18T14:42:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d3a756640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d340ce040 0x7f7d340cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d39f55640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7d3415bd40 0x7f7d340d1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41570/0 (socket says 172.21.15.6:41570) 2024-03-18T14:42:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d39f55640 1 -- 172.21.15.6:0/3448125017 learned_addr learned my addr 172.21.15.6:0/3448125017 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.759+0000 7f7d3a756640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d340ce040 0x7f7d340cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57920/0 (socket says 172.21.15.6:57920) 2024-03-18T14:42:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.760+0000 7f7d3a756640 1 -- 172.21.15.6:0/3448125017 >> v1:172.21.15.92:6789/0 conn(0x7f7d340d1f60 legacy=0x7f7d340cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.760+0000 7f7d3a756640 1 -- 172.21.15.6:0/3448125017 >> v1:172.21.15.6:6789/0 conn(0x7f7d3415bd40 legacy=0x7f7d340d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.760+0000 7f7d38f53640 1 -- 172.21.15.6:0/3448125017 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597198463 0 0) 0x7f7d340cc870 con 0x7f7d3415bd40 2024-03-18T14:42:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.760+0000 7f7d3a756640 1 -- 172.21.15.6:0/3448125017 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d340ccb10 con 0x7f7d340ce040 2024-03-18T14:42:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.760+0000 7f7d3a756640 1 --2- 172.21.15.6:0/3448125017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d340ce040 0x7f7d340cfea0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097a0 tx=0x7f7d2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693dd0690d85ad8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d38f53640 1 -- 172.21.15.6:0/3448125017 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c035020 con 0x7f7d340ce040 2024-03-18T14:42:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d38f53640 1 -- 172.21.15.6:0/3448125017 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2c031830 con 0x7f7d340ce040 2024-03-18T14:42:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d38f53640 1 -- 172.21.15.6:0/3448125017 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c031c00 con 0x7f7d340ce040 2024-03-18T14:42:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/3448125017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d340ce040 msgr2=0x7f7d340cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/3448125017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d340ce040 0x7f7d340cfea0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097a0 tx=0x7f7d2c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/3448125017 shutdown_connections 2024-03-18T14:42:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/3448125017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d340ce040 0x7f7d340cfea0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/3448125017 >> 172.21.15.6:0/3448125017 conn(0x7f7d3409ae20 msgr2=0x7f7d3408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/3448125017 shutdown_connections 2024-03-18T14:42:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.761+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/3448125017 wait complete. 2024-03-18T14:42:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.762+0000 7f7d3c1b9640 1 Processor -- start 2024-03-18T14:42:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.762+0000 7f7d3c1b9640 1 -- start start 2024-03-18T14:42:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3c1b9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d340ce040 0x7f7d340c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3c1b9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d340d1f60 0x7f7d340c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3c1b9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d3415bd40 0x7f7d340b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3c1b9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3415c2a0 con 0x7f7d3415bd40 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3c1b9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3415c120 con 0x7f7d340d1f60 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3c1b9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7d34166a00 con 0x7f7d340ce040 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d3415bd40 0x7f7d340b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d3415bd40 0x7f7d340b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57934/0 (socket says 172.21.15.6:57934) 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 -- 172.21.15.6:0/2897716233 learned_addr learned my addr 172.21.15.6:0/2897716233 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d39754640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d340d1f60 0x7f7d340c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d39f55640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d340ce040 0x7f7d340c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 -- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d340ce040 msgr2=0x7f7d340c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d340ce040 0x7f7d340c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 -- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d340d1f60 msgr2=0x7f7d340c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d340d1f60 0x7f7d340c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.763+0000 7f7d3a756640 1 -- 172.21.15.6:0/2897716233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d340aee10 con 0x7f7d3415bd40 2024-03-18T14:42:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.764+0000 7f7d3a756640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d3415bd40 0x7f7d340b0640 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c034040 tx=0x7f7d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.764+0000 7f7d327fc640 1 -- 172.21.15.6:0/2897716233 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c041070 con 0x7f7d3415bd40 2024-03-18T14:42:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.764+0000 7f7d327fc640 1 -- 172.21.15.6:0/2897716233 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2c004250 con 0x7f7d3415bd40 2024-03-18T14:42:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.764+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d340b10c0 con 0x7f7d3415bd40 2024-03-18T14:42:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.764+0000 7f7d327fc640 1 -- 172.21.15.6:0/2897716233 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c035040 con 0x7f7d3415bd40 2024-03-18T14:42:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.764+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d340ae8f0 con 0x7f7d3415bd40 2024-03-18T14:42:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.765+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d340ae710 con 0x7f7d3415bd40 2024-03-18T14:42:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.765+0000 7f7d327fc640 1 -- 172.21.15.6:0/2897716233 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7d2c038020 con 0x7f7d3415bd40 2024-03-18T14:42:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.766+0000 7f7d327fc640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.766+0000 7f7d327fc640 1 -- 172.21.15.6:0/2897716233 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f7d2c0ac650 con 0x7f7d3415bd40 2024-03-18T14:42:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.766+0000 7f7d39f55640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.766+0000 7f7d39f55640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7d24005fd0 tx=0x7f7d24007f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.770+0000 7f7d327fc640 1 -- 172.21.15.6:0/2897716233 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7d340ae710 con 0x7f7d3415bd40 2024-03-18T14:42:57.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.952+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d340aeaf0 con 0x7f7d3415bd40 2024-03-18T14:42:57.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.953+0000 7f7d327fc640 1 -- 172.21.15.6:0/2897716233 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16859 (secure 0 0 0) 0x7f7d340aeaf0 con 0x7f7d3415bd40 2024-03-18T14:42:57.955 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:57.955 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:56.672498+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 msgr2=0x7f7d2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7d24005fd0 tx=0x7f7d24007f10 comp rx=0 tx=0).stop 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d3415bd40 msgr2=0x7f7d340b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d3415bd40 0x7f7d340b0640 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c034040 tx=0x7f7d2c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 shutdown_connections 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d3415bd40 0x7f7d340b0640 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d340d1f60 0x7f7d340c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 --2- 172.21.15.6:0/2897716233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d340ce040 0x7f7d340c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 >> 172.21.15.6:0/2897716233 conn(0x7f7d3409ae20 msgr2=0x7f7d3415a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 shutdown_connections 2024-03-18T14:42:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:57.955+0000 7f7d3c1b9640 1 -- 172.21.15.6:0/2897716233 wait complete. 2024-03-18T14:42:57.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T14:42:58.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.070+0000 7fe3f490c640 1 Processor -- start 2024-03-18T14:42:58.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.070+0000 7fe3f490c640 1 -- start start 2024-03-18T14:42:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3f490c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f00b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3f490c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3f00b5740 0x7fe3f00b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3f490c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3f015e3b0 con 0x7fe3f00d1e40 2024-03-18T14:42:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3f490c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe3f00b60e0 con 0x7fe3f00b5740 2024-03-18T14:42:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3f490c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f00b8470 con 0x7fe3f015dfd0 2024-03-18T14:42:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3eed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe3f00d1e40 0x7fe3f00d22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41580/0 (socket says 172.21.15.6:41580) 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3eed9d640 1 -- 172.21.15.6:0/2876635316 learned_addr learned my addr 172.21.15.6:0/2876635316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3edd9b640 1 --2- 172.21.15.6:0/2876635316 >> v2:172.21.15.92:3300/0 conn(0x7fe3f00b5740 0x7fe3f00b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.071+0000 7fe3ee59c640 1 --2- 172.21.15.6:0/2876635316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f00b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.072+0000 7fe3ed59a640 1 -- 172.21.15.6:0/2876635316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491753451 0 0) 0x7fe3f015e3b0 con 0x7fe3f00d1e40 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.072+0000 7fe3ed59a640 1 -- 172.21.15.6:0/2876635316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3dc003520 con 0x7fe3f00d1e40 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.072+0000 7fe3ee59c640 1 -- 172.21.15.6:0/2876635316 >> v2:172.21.15.92:3300/0 conn(0x7fe3f00b5740 msgr2=0x7fe3f00b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.072+0000 7fe3ee59c640 1 --2- 172.21.15.6:0/2876635316 >> v2:172.21.15.92:3300/0 conn(0x7fe3f00b5740 0x7fe3f00b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.072+0000 7fe3ee59c640 1 -- 172.21.15.6:0/2876635316 >> v1:172.21.15.6:6789/0 conn(0x7fe3f00d1e40 legacy=0x7fe3f00d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.072+0000 7fe3ee59c640 1 -- 172.21.15.6:0/2876635316 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f00b8750 con 0x7fe3f015dfd0 2024-03-18T14:42:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.072+0000 7fe3edd9b640 1 --2- 172.21.15.6:0/2876635316 >> v2:172.21.15.92:3300/0 conn(0x7fe3f00b5740 0x7fe3f00b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:42:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3ee59c640 1 --2- 172.21.15.6:0/2876635316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f00b9650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3e80097a0 tx=0x7fe3e802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f6394a227dc698 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3ed59a640 1 -- 172.21.15.6:0/2876635316 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8035020 con 0x7fe3f015dfd0 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3ed59a640 1 -- 172.21.15.6:0/2876635316 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e8031810 con 0x7fe3f015dfd0 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3ed59a640 1 -- 172.21.15.6:0/2876635316 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8031bc0 con 0x7fe3f015dfd0 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 -- 172.21.15.6:0/2876635316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 msgr2=0x7fe3f00b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 --2- 172.21.15.6:0/2876635316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f00b9650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3e80097a0 tx=0x7fe3e802b8b0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 -- 172.21.15.6:0/2876635316 shutdown_connections 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 --2- 172.21.15.6:0/2876635316 >> v2:172.21.15.92:3300/0 conn(0x7fe3f00b5740 0x7fe3f00b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 --2- 172.21.15.6:0/2876635316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f00b9650 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 -- 172.21.15.6:0/2876635316 >> 172.21.15.6:0/2876635316 conn(0x7fe3f0072d10 msgr2=0x7fe3f0073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 -- 172.21.15.6:0/2876635316 shutdown_connections 2024-03-18T14:42:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.073+0000 7fe3f490c640 1 -- 172.21.15.6:0/2876635316 wait complete. 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.075+0000 7fe3f490c640 1 Processor -- start 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.075+0000 7fe3f490c640 1 -- start start 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.075+0000 7fe3f490c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f00b5740 0x7fe3f00c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.075+0000 7fe3f490c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f00d1e40 0x7fe3f015fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.075+0000 7fe3f490c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f015bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.075+0000 7fe3f490c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f00b9b90 con 0x7fe3f00d1e40 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3f490c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f00b8d10 con 0x7fe3f00b5740 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3f490c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f00b7e00 con 0x7fe3f015dfd0 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f00d1e40 0x7fe3f015fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f00d1e40 0x7fe3f015fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57950/0 (socket says 172.21.15.6:57950) 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 -- 172.21.15.6:0/3113231511 learned_addr learned my addr 172.21.15.6:0/3113231511 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3ee59c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f00b5740 0x7fe3f00c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3eed9d640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f015bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 -- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 msgr2=0x7fe3f015bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f015bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 -- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f00b5740 msgr2=0x7fe3f00c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f00b5740 0x7fe3f00c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 -- 172.21.15.6:0/3113231511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f015eaf0 con 0x7fe3f00d1e40 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3edd9b640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f00d1e40 0x7fe3f015fb90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800ec10 tx=0x7fe3d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3e6ffd640 1 -- 172.21.15.6:0/3113231511 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8017070 con 0x7fe3f00d1e40 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3e6ffd640 1 -- 172.21.15.6:0/3113231511 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d800cc30 con 0x7fe3f00d1e40 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3e6ffd640 1 -- 172.21.15.6:0/3113231511 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8013470 con 0x7fe3f00d1e40 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f015edd0 con 0x7fe3f00d1e40 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.076+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f015f390 con 0x7fe3f00d1e40 2024-03-18T14:42:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.077+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f00751b0 con 0x7fe3f00d1e40 2024-03-18T14:42:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.078+0000 7fe3e6ffd640 1 -- 172.21.15.6:0/3113231511 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe3d801a070 con 0x7fe3f00d1e40 2024-03-18T14:42:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.078+0000 7fe3e6ffd640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d006bbd0 0x7fe3d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.078+0000 7fe3e6ffd640 1 -- 172.21.15.6:0/3113231511 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe3d808add0 con 0x7fe3f00d1e40 2024-03-18T14:42:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.078+0000 7fe3ee59c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d006bbd0 0x7fe3d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.079+0000 7fe3ee59c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d006bbd0 0x7fe3d006e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8036000 tx=0x7fe3e803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:58.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.083+0000 7fe3e6ffd640 1 -- 172.21.15.6:0/3113231511 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe3d8059170 con 0x7fe3f00d1e40 2024-03-18T14:42:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.303+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe3f00008d0 con 0x7fe3f00d1e40 2024-03-18T14:42:58.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.317+0000 7fe3e6ffd640 1 -- 172.21.15.6:0/3113231511 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7fe3d805d210 con 0x7fe3f00d1e40 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d006bbd0 msgr2=0x7fe3d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d006bbd0 0x7fe3d006e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8036000 tx=0x7fe3e803c040 comp rx=0 tx=0).stop 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f00d1e40 msgr2=0x7fe3f015fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f00d1e40 0x7fe3f015fb90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800ec10 tx=0x7fe3d800c520 comp rx=0 tx=0).stop 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 shutdown_connections 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3d006bbd0 0x7fe3d006e090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3f015dfd0 0x7fe3f015bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3f00d1e40 0x7fe3f015fb90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 --2- 172.21.15.6:0/3113231511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3f00b5740 0x7fe3f00c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 >> 172.21.15.6:0/3113231511 conn(0x7fe3f0072d10 msgr2=0x7fe3f00c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:58.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 shutdown_connections 2024-03-18T14:42:58.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.319+0000 7fe3f490c640 1 -- 172.21.15.6:0/3113231511 wait complete. 2024-03-18T14:42:58.331 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T14:42:58.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T14:42:58.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.435+0000 7f1532fe6640 1 Processor -- start 2024-03-18T14:42:58.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.435+0000 7f1532fe6640 1 -- start start 2024-03-18T14:42:58.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1532fe6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f152c0d0b60 0x7f152c0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1532fe6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f152c0d39d0 con 0x7f152c15e030 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1532fe6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f152c0d2fd0 con 0x7f152c0d3570 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1532fe6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f152c15e410 con 0x7f152c0d0b60 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1530d82640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f152c15e030 0x7f152c0ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41582/0 (socket says 172.21.15.6:41582) 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1530d82640 1 -- 172.21.15.6:0/2168270499 learned_addr learned my addr 172.21.15.6:0/2168270499 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1531583640 1 --2- 172.21.15.6:0/2168270499 >> v2:172.21.15.92:3300/0 conn(0x7f152c0d0b60 0x7f152c0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f15237fe640 1 -- 172.21.15.6:0/2168270499 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1544685347 0 0) 0x7f152c0d39d0 con 0x7f152c15e030 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f15237fe640 1 -- 172.21.15.6:0/2168270499 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1514003520 con 0x7f152c15e030 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f15237fe640 1 -- 172.21.15.6:0/2168270499 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3830063310 0 0) 0x7f152c0d2fd0 con 0x7f152c0d3570 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f15237fe640 1 -- 172.21.15.6:0/2168270499 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f152c0d39d0 con 0x7f152c0d3570 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1531583640 1 -- 172.21.15.6:0/2168270499 >> v1:172.21.15.92:6789/0 conn(0x7f152c0d3570 legacy=0x7f152c0d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1531583640 1 -- 172.21.15.6:0/2168270499 >> v1:172.21.15.6:6789/0 conn(0x7f152c15e030 legacy=0x7f152c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.436+0000 7f1531583640 1 -- 172.21.15.6:0/2168270499 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152c0d3290 con 0x7f152c0d0b60 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.437+0000 7f1531583640 1 --2- 172.21.15.6:0/2168270499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c0b6140 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1524009d90 tx=0x7f152402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f0252d02738183 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.437+0000 7f15237fe640 1 -- 172.21.15.6:0/2168270499 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524039070 con 0x7f152c0d0b60 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.437+0000 7f15237fe640 1 -- 172.21.15.6:0/2168270499 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f152402cd00 con 0x7f152c0d0b60 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.437+0000 7f15237fe640 1 -- 172.21.15.6:0/2168270499 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524034660 con 0x7f152c0d0b60 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.438+0000 7f1532fe6640 1 -- 172.21.15.6:0/2168270499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 msgr2=0x7f152c0b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.438+0000 7f1532fe6640 1 --2- 172.21.15.6:0/2168270499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c0b6140 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1524009d90 tx=0x7f152402bf50 comp rx=0 tx=0).stop 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.438+0000 7f1532fe6640 1 -- 172.21.15.6:0/2168270499 shutdown_connections 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.438+0000 7f1532fe6640 1 --2- 172.21.15.6:0/2168270499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c0b6140 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.438+0000 7f1532fe6640 1 -- 172.21.15.6:0/2168270499 >> 172.21.15.6:0/2168270499 conn(0x7f152c06c8c0 msgr2=0x7f152c06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.438+0000 7f1532fe6640 1 -- 172.21.15.6:0/2168270499 shutdown_connections 2024-03-18T14:42:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.438+0000 7f1532fe6640 1 -- 172.21.15.6:0/2168270499 wait complete. 2024-03-18T14:42:58.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.439+0000 7f1532fe6640 1 Processor -- start 2024-03-18T14:42:58.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.439+0000 7f1532fe6640 1 -- start start 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1532fe6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c1550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1532fe6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f152c0d3570 0x7f152c0b8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1532fe6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f152c15e030 0x7f152c0b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1532fe6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f152c0ce830 con 0x7f152c0d3570 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1532fe6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f152c0ce300 con 0x7f152c0d0b60 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1532fe6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f152c161a90 con 0x7f152c15e030 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f152c0d3570 0x7f152c0b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f152c0d3570 0x7f152c0b8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57956/0 (socket says 172.21.15.6:57956) 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 -- 172.21.15.6:0/3728170143 learned_addr learned my addr 172.21.15.6:0/3728170143 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1531583640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f152c15e030 0x7f152c0b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1530d82640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c1550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 -- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f152c15e030 msgr2=0x7f152c0b85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f152c15e030 0x7f152c0b85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 -- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 msgr2=0x7f152c1550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c1550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1523fff640 1 -- 172.21.15.6:0/3728170143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152c1f7fc0 con 0x7f152c0d3570 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.440+0000 7f1530d82640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c1550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.441+0000 7f1523fff640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f152c0d3570 0x7f152c0b8070 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f151c00c500 tx=0x7f151c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.441+0000 7f15217fa640 1 -- 172.21.15.6:0/3728170143 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c0052d0 con 0x7f152c0d3570 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.441+0000 7f15217fa640 1 -- 172.21.15.6:0/3728170143 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f151c005490 con 0x7f152c0d3570 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.441+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f152c1f73b0 con 0x7f152c0d3570 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.441+0000 7f15217fa640 1 -- 172.21.15.6:0/3728170143 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c012870 con 0x7f152c0d3570 2024-03-18T14:42:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.441+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f152c1f7970 con 0x7f152c0d3570 2024-03-18T14:42:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.442+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f152c0b75d0 con 0x7f152c0d3570 2024-03-18T14:42:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.442+0000 7f15217fa640 1 -- 172.21.15.6:0/3728170143 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f151c012a10 con 0x7f152c0d3570 2024-03-18T14:42:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.443+0000 7f15217fa640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f151806bbd0 0x7f151806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.443+0000 7f15217fa640 1 -- 172.21.15.6:0/3728170143 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f151c08a950 con 0x7f152c0d3570 2024-03-18T14:42:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.443+0000 7f1530d82640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f151806bbd0 0x7f151806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.443+0000 7f1530d82640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f151806bbd0 0x7f151806e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f15280011e0 tx=0x7f1528009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.451+0000 7f15217fa640 1 -- 172.21.15.6:0/3728170143 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f152c0b75d0 con 0x7f152c0d3570 2024-03-18T14:42:58.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:58.671+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f152c15fac0 con 0x7f152c0d3570 2024-03-18T14:42:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.321+0000 7f15217fa640 1 -- 172.21.15.6:0/3728170143 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f151c059220 con 0x7f152c0d3570 2024-03-18T14:42:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f151806bbd0 msgr2=0x7f151806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f151806bbd0 0x7f151806e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f15280011e0 tx=0x7f1528009040 comp rx=0 tx=0).stop 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f152c0d3570 msgr2=0x7f152c0b8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f152c0d3570 0x7f152c0b8070 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f151c00c500 tx=0x7f151c00c9c0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 shutdown_connections 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f151806bbd0 0x7f151806e090 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f152c15e030 0x7f152c0b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f152c0d3570 0x7f152c0b8070 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 --2- 172.21.15.6:0/3728170143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f152c0d0b60 0x7f152c1550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 >> 172.21.15.6:0/3728170143 conn(0x7f152c06c8c0 msgr2=0x7f152c15d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 shutdown_connections 2024-03-18T14:42:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.323+0000 7f1532fe6640 1 -- 172.21.15.6:0/3728170143 wait complete. 2024-03-18T14:42:59.335 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T14:42:59.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:42:59.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.436+0000 7f4be110c640 1 Processor -- start 2024-03-18T14:42:59.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.436+0000 7f4be110c640 1 -- start start 2024-03-18T14:42:59.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.437+0000 7f4be110c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4bdc0ad700 0x7f4bdc0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.437+0000 7f4be110c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bdc0cb5b0 con 0x7f4bdc0d09a0 2024-03-18T14:42:59.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.437+0000 7f4be110c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bdc0cb850 con 0x7f4bdc0b0e80 2024-03-18T14:42:59.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.437+0000 7f4be110c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4bdc0b1a50 con 0x7f4bdc0ad700 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bdad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4bdc0d09a0 0x7f4bdc0d0d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41598/0 (socket says 172.21.15.6:41598) 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bdad9d640 1 -- 172.21.15.6:0/4186818823 learned_addr learned my addr 172.21.15.6:0/4186818823 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bda59c640 1 --2- 172.21.15.6:0/4186818823 >> v2:172.21.15.92:3300/0 conn(0x7f4bdc0ad700 0x7f4bdc0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951965086 0 0) 0x7f4bdc0cb5b0 con 0x7f4bdc0d09a0 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bc4003520 con 0x7f4bdc0d09a0 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 493889407 0 0) 0x7f4bc4003520 con 0x7f4bdc0d09a0 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 >> v2:172.21.15.92:3300/0 conn(0x7f4bdc0ad700 msgr2=0x7f4bdc0b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 --2- 172.21.15.6:0/4186818823 >> v2:172.21.15.92:3300/0 conn(0x7f4bdc0ad700 0x7f4bdc0b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 >> v1:172.21.15.92:6789/0 conn(0x7f4bdc0b0e80 legacy=0x7f4bdc0cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bdc15b070 con 0x7f4bdc0d09a0 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f4bcc001670 con 0x7f4bdc0d09a0 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 >> v1:172.21.15.6:6789/0 conn(0x7f4bdc0d09a0 legacy=0x7f4bdc0d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bc40039b0 0x7f4bc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bc40043b0 0x7f4bc4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc0b1a50 con 0x7f4bc4005080 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc400bfe0 con 0x7f4bc40043b0 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4bc400c1e0 con 0x7f4bc40039b0 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bdb59e640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bdad9d640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bc40039b0 0x7f4bc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.438+0000 7f4bda59c640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bc40043b0 0x7f4bc4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.439+0000 7f4bdb59e640 1 -- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bc40039b0 msgr2=0x7f4bc4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.439+0000 7f4bdb59e640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bc40039b0 0x7f4bc4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.439+0000 7f4bdb59e640 1 -- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bc40043b0 msgr2=0x7f4bc4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.439+0000 7f4bdb59e640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bc40043b0 0x7f4bc4004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.439+0000 7f4bdb59e640 1 -- 172.21.15.6:0/4186818823 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc4003520 con 0x7f4bc4005080 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4bdb59e640 1 -- 172.21.15.6:0/4186818823 reap_dead start 2024-03-18T14:42:59.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4bdb59e640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bc400b9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4001f60 tx=0x7f4bc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38f7c46141527bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:59.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4001b30 con 0x7f4bc4005080 2024-03-18T14:42:59.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd4003b90 con 0x7f4bc4005080 2024-03-18T14:42:59.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4bd9d9b640 1 -- 172.21.15.6:0/4186818823 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4005070 con 0x7f4bc4005080 2024-03-18T14:42:59.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4be110c640 1 -- 172.21.15.6:0/4186818823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 msgr2=0x7f4bc400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4be110c640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bc400b9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4001f60 tx=0x7f4bc4005830 comp rx=0 tx=0).stop 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4be110c640 1 -- 172.21.15.6:0/4186818823 shutdown_connections 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.440+0000 7f4be110c640 1 --2- 172.21.15.6:0/4186818823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bc400b9c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.441+0000 7f4be110c640 1 -- 172.21.15.6:0/4186818823 >> 172.21.15.6:0/4186818823 conn(0x7f4bdc099a50 msgr2=0x7f4bdc099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.441+0000 7f4be110c640 1 -- 172.21.15.6:0/4186818823 shutdown_connections 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.441+0000 7f4be110c640 1 -- 172.21.15.6:0/4186818823 wait complete. 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.442+0000 7f4be110c640 1 Processor -- start 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.442+0000 7f4be110c640 1 -- start start 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4be110c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bdc0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4be110c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bdc1558d0 0x7f4bdc155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4be110c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bdc146f30 0x7f4bdc1473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4be110c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc0b1490 con 0x7f4bc4005080 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4be110c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc0cbb80 con 0x7f4bdc1558d0 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4be110c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4bdc0b1770 con 0x7f4bdc146f30 2024-03-18T14:42:59.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bdc0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bdc0b60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57972/0 (socket says 172.21.15.6:57972) 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 -- 172.21.15.6:0/3656570892 learned_addr learned my addr 172.21.15.6:0/3656570892 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdb59e640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bdc146f30 0x7f4bdc1473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bda59c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bdc1558d0 0x7f4bdc155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 -- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bdc146f30 msgr2=0x7f4bdc1473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bdc146f30 0x7f4bdc1473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 -- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bdc1558d0 msgr2=0x7f4bdc155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bdc1558d0 0x7f4bdc155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 -- 172.21.15.6:0/3656570892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bdc156390 con 0x7f4bc4005080 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bda59c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bdc1558d0 0x7f4bdc155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.443+0000 7f4bdad9d640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bdc0b60d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc007310 tx=0x7f4bcc0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.444+0000 7f4bc37fe640 1 -- 172.21.15.6:0/3656570892 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc01a070 con 0x7f4bc4005080 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.444+0000 7f4bc37fe640 1 -- 172.21.15.6:0/3656570892 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc00f8e0 con 0x7f4bc4005080 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.444+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bdc156670 con 0x7f4bc4005080 2024-03-18T14:42:59.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.444+0000 7f4bc37fe640 1 -- 172.21.15.6:0/3656570892 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00fcb0 con 0x7f4bc4005080 2024-03-18T14:42:59.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.444+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bdc156c30 con 0x7f4bc4005080 2024-03-18T14:42:59.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.445+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bdc09ea80 con 0x7f4bc4005080 2024-03-18T14:42:59.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.445+0000 7f4bc37fe640 1 -- 172.21.15.6:0/3656570892 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4bcc01d070 con 0x7f4bc4005080 2024-03-18T14:42:59.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.446+0000 7f4bc37fe640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4bc806bc00 0x7f4bc806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.446+0000 7f4bc37fe640 1 -- 172.21.15.6:0/3656570892 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4bcc08a380 con 0x7f4bc4005080 2024-03-18T14:42:59.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.446+0000 7f4bda59c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4bc806bc00 0x7f4bc806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.446+0000 7f4bda59c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4bc806bc00 0x7f4bc806e0c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0cb040 tx=0x7f4bd0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:59.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.450+0000 7f4bc37fe640 1 -- 172.21.15.6:0/3656570892 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4bcc058720 con 0x7f4bc4005080 2024-03-18T14:42:59.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.672+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bdc04f620 con 0x7f4bc4005080 2024-03-18T14:42:59.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.672+0000 7f4bc37fe640 1 -- 172.21.15.6:0/3656570892 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5169 (secure 0 0 0) 0x7f4bcc05c7c0 con 0x7f4bc4005080 2024-03-18T14:42:59.674 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:59.674 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:42:56.672813+0000","modified":"2024-03-18T14:42:58.672583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5529":{"gid":5529,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2677248951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2677248951},{"type":"v1","addr":"172.21.15.6:6834","nonce":2677248951}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5529,"qdb_cluster":[5529]},"id":2}]} 2024-03-18T14:42:59.674 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 15 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4bc806bc00 msgr2=0x7f4bc806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4bc806bc00 0x7f4bc806e0c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0cb040 tx=0x7f4bd0008040 comp rx=0 tx=0).stop 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 msgr2=0x7f4bdc0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bdc0b60d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc007310 tx=0x7f4bcc0077d0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 shutdown_connections 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4bc806bc00 0x7f4bc806e0c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bdc146f30 0x7f4bdc1473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bdc1558d0 0x7f4bdc155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 --2- 172.21.15.6:0/3656570892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4005080 0x7f4bdc0b60d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 >> 172.21.15.6:0/3656570892 conn(0x7f4bdc099a50 msgr2=0x7f4bdc0d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:59.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 shutdown_connections 2024-03-18T14:42:59.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.674+0000 7f4be110c640 1 -- 172.21.15.6:0/3656570892 wait complete. 2024-03-18T14:42:59.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:42:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.787+0000 7f4a2eac6640 1 Processor -- start 2024-03-18T14:42:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.787+0000 7f4a2eac6640 1 -- start start 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2eac6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a2815e020 0x7f4a28162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2eac6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a280b9660 con 0x7f4a280ced00 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2eac6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a280ce390 con 0x7f4a280d3210 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2eac6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4a280d2270 con 0x7f4a2815e020 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2c862640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a2815e020 0x7f4a28162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a27fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4a280ced00 0x7f4a280d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41610/0 (socket says 172.21.15.6:41610) 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2c862640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a2815e020 0x7f4a28162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57984/0 (socket says 172.21.15.6:57984) 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a27fff640 1 -- 172.21.15.6:0/2658795458 learned_addr learned my addr 172.21.15.6:0/2658795458 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2c862640 1 -- 172.21.15.6:0/2658795458 >> v1:172.21.15.92:6789/0 conn(0x7f4a280d3210 legacy=0x7f4a280d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2c862640 1 -- 172.21.15.6:0/2658795458 >> v1:172.21.15.6:6789/0 conn(0x7f4a280ced00 legacy=0x7f4a280d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a277fe640 1 -- 172.21.15.6:0/2658795458 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775236041 0 0) 0x7f4a280b9660 con 0x7f4a280ced00 2024-03-18T14:42:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.788+0000 7f4a2c862640 1 -- 172.21.15.6:0/2658795458 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a280b9900 con 0x7f4a2815e020 2024-03-18T14:42:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.789+0000 7f4a2c862640 1 --2- 172.21.15.6:0/2658795458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a2815e020 0x7f4a28162380 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0097a0 tx=0x7f4a1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d35680ac5af1a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.789+0000 7f4a277fe640 1 -- 172.21.15.6:0/2658795458 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c035020 con 0x7f4a2815e020 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.789+0000 7f4a277fe640 1 -- 172.21.15.6:0/2658795458 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a1c031830 con 0x7f4a2815e020 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.789+0000 7f4a277fe640 1 -- 172.21.15.6:0/2658795458 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c031c00 con 0x7f4a2815e020 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.790+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2658795458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a2815e020 msgr2=0x7f4a28162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.790+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2658795458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a2815e020 0x7f4a28162380 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0097a0 tx=0x7f4a1c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.790+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2658795458 shutdown_connections 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.790+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2658795458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a2815e020 0x7f4a28162380 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.790+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2658795458 >> 172.21.15.6:0/2658795458 conn(0x7f4a28072ce0 msgr2=0x7f4a280730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.790+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2658795458 shutdown_connections 2024-03-18T14:42:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.790+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2658795458 wait complete. 2024-03-18T14:42:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.791+0000 7f4a2eac6640 1 Processor -- start 2024-03-18T14:42:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.791+0000 7f4a2eac6640 1 -- start start 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2eac6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a280ced00 0x7f4a28152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2eac6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a280d3210 0x7f4a2815c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2eac6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2815e020 0x7f4a2815c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2eac6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280cf2c0 con 0x7f4a280ced00 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2eac6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280ce890 con 0x7f4a2815e020 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2eac6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4a280cf440 con 0x7f4a280d3210 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a280ced00 0x7f4a28152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a280ced00 0x7f4a28152120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57998/0 (socket says 172.21.15.6:57998) 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 -- 172.21.15.6:0/2917788401 learned_addr learned my addr 172.21.15.6:0/2917788401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a27fff640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a280d3210 0x7f4a2815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2d063640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2815e020 0x7f4a2815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 -- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a280d3210 msgr2=0x7f4a2815c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a280d3210 0x7f4a2815c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 -- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2815e020 msgr2=0x7f4a2815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2815e020 0x7f4a2815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2c862640 1 -- 172.21.15.6:0/2917788401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2815e680 con 0x7f4a280ced00 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a2d063640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2815e020 0x7f4a2815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.792+0000 7f4a27fff640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a280d3210 0x7f4a2815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.793+0000 7f4a2c862640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a280ced00 0x7f4a28152120 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0097a0 tx=0x7f4a1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.793+0000 7f4a257fa640 1 -- 172.21.15.6:0/2917788401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c041070 con 0x7f4a280ced00 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.793+0000 7f4a257fa640 1 -- 172.21.15.6:0/2917788401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a1c004230 con 0x7f4a280ced00 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.793+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2815e960 con 0x7f4a280ced00 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.793+0000 7f4a257fa640 1 -- 172.21.15.6:0/2917788401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c035040 con 0x7f4a280ced00 2024-03-18T14:42:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.793+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2815ef20 con 0x7f4a280ced00 2024-03-18T14:42:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.794+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2808afe0 con 0x7f4a280ced00 2024-03-18T14:42:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.794+0000 7f4a257fa640 1 -- 172.21.15.6:0/2917788401 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4a1c038020 con 0x7f4a280ced00 2024-03-18T14:42:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.795+0000 7f4a257fa640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4a0c06bbd0 0x7f4a0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:42:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.795+0000 7f4a257fa640 1 -- 172.21.15.6:0/2917788401 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4a1c0ac640 con 0x7f4a280ced00 2024-03-18T14:42:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.795+0000 7f4a27fff640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4a0c06bbd0 0x7f4a0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:42:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.796+0000 7f4a27fff640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4a0c06bbd0 0x7f4a0c06e090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4a2815ccc0 tx=0x7f4a14009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:42:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.799+0000 7f4a257fa640 1 -- 172.21.15.6:0/2917788401 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4a1c07b030 con 0x7f4a280ced00 2024-03-18T14:42:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.982+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a2804f620 con 0x7f4a280ced00 2024-03-18T14:42:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.983+0000 7f4a257fa640 1 -- 172.21.15.6:0/2917788401 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16846 (secure 0 0 0) 0x7f4a1c07a930 con 0x7f4a280ced00 2024-03-18T14:42:59.984 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:42:59.985 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":39,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:42:58.311250+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:42:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4a0c06bbd0 msgr2=0x7f4a0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4a0c06bbd0 0x7f4a0c06e090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4a2815ccc0 tx=0x7f4a14009040 comp rx=0 tx=0).stop 2024-03-18T14:42:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a280ced00 msgr2=0x7f4a28152120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:42:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a280ced00 0x7f4a28152120 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0097a0 tx=0x7f4a1c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 shutdown_connections 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4a0c06bbd0 0x7f4a0c06e090 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2815e020 0x7f4a2815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a280d3210 0x7f4a2815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 --2- 172.21.15.6:0/2917788401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a280ced00 0x7f4a28152120 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 >> 172.21.15.6:0/2917788401 conn(0x7f4a28072ce0 msgr2=0x7f4a280cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 shutdown_connections 2024-03-18T14:42:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:42:59.985+0000 7f4a2eac6640 1 -- 172.21.15.6:0/2917788401 wait complete. 2024-03-18T14:42:59.997 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T14:42:59.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:43:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.104+0000 7fa314cc6640 1 Processor -- start 2024-03-18T14:43:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.104+0000 7fa314cc6640 1 -- start start 2024-03-18T14:43:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa314cc6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 0x7fa310166850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa314cc6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3100b1c10 con 0x7fa3100d17f0 2024-03-18T14:43:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa314cc6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3100b1eb0 con 0x7fa3100b17b0 2024-03-18T14:43:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa314cc6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa3100b2190 con 0x7fa3100cd9a0 2024-03-18T14:43:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa30e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3100d17f0 0x7fa3100d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41620/0 (socket says 172.21.15.6:41620) 2024-03-18T14:43:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa30e59c640 1 -- 172.21.15.6:0/1263416964 learned_addr learned my addr 172.21.15.6:0/1263416964 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa30ed9d640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 0x7fa310166850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1935934487 0 0) 0x7fa3100b1c10 con 0x7fa3100d17f0 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.106+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2fc003520 con 0x7fa3100d17f0 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1994582462 0 0) 0x7fa2fc003520 con 0x7fa3100d17f0 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 msgr2=0x7fa310166850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 0x7fa310166850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 >> v1:172.21.15.92:6789/0 conn(0x7fa3100b17b0 legacy=0x7fa3100d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3100b2470 con 0x7fa3100d17f0 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fa308001690 con 0x7fa3100d17f0 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 >> v1:172.21.15.6:6789/0 conn(0x7fa3100d17f0 legacy=0x7fa3100d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc0039e0 0x7fa2fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc004800 0x7fa2fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 0x7fa2fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa3100b2190 con 0x7fa2fc0039e0 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc00baa0 con 0x7fa2fc004800 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc00bca0 con 0x7fa3100cd9a0 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30e59c640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc0039e0 0x7fa2fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30ed9d640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 0x7fa2fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30dd9b640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc004800 0x7fa2fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30e59c640 1 -- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 msgr2=0x7fa2fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30e59c640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100cd9a0 0x7fa2fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30e59c640 1 -- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc004800 msgr2=0x7fa2fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30e59c640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc004800 0x7fa2fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.107+0000 7fa30e59c640 1 -- 172.21.15.6:0/1263416964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc003520 con 0x7fa2fc0039e0 2024-03-18T14:43:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.108+0000 7fa30e59c640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc0039e0 0x7fa2fc003e00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa3100b1de0 tx=0x7fa2fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aca1a1ba9c943d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.108+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308007d60 con 0x7fa2fc0039e0 2024-03-18T14:43:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.108+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa308030070 con 0x7fa2fc0039e0 2024-03-18T14:43:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.108+0000 7fa30d59a640 1 -- 172.21.15.6:0/1263416964 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308005d30 con 0x7fa2fc0039e0 2024-03-18T14:43:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.109+0000 7fa314cc6640 1 -- 172.21.15.6:0/1263416964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc0039e0 msgr2=0x7fa2fc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.109+0000 7fa314cc6640 1 --2- 172.21.15.6:0/1263416964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc0039e0 0x7fa2fc003e00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa3100b1de0 tx=0x7fa2fc005830 comp rx=0 tx=0).stop 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.109+0000 7fa30ed9d640 1 -- 172.21.15.6:0/1263416964 reap_dead start 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.109+0000 7fa314cc6640 1 -- 172.21.15.6:0/1263416964 shutdown_connections 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.109+0000 7fa314cc6640 1 -- 172.21.15.6:0/1263416964 >> 172.21.15.6:0/1263416964 conn(0x7fa31008b130 msgr2=0x7fa31008b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.109+0000 7fa314cc6640 1 -- 172.21.15.6:0/1263416964 shutdown_connections 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.109+0000 7fa314cc6640 1 -- 172.21.15.6:0/1263416964 wait complete. 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.110+0000 7fa314cc6640 1 Processor -- start 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.110+0000 7fa314cc6640 1 -- start start 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa314cc6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc0039e0 0x7fa3100c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa314cc6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100c7c00 0x7fa310153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa314cc6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3100c5670 0x7fa310163bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa314cc6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa31015c080 con 0x7fa3100c5670 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa314cc6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3100b2a30 con 0x7fa2fc0039e0 2024-03-18T14:43:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa314cc6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa310164840 con 0x7fa3100c7c00 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa30ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3100c5670 0x7fa310163bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa30ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3100c5670 0x7fa310163bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58002/0 (socket says 172.21.15.6:58002) 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa30ed9d640 1 -- 172.21.15.6:0/2472201928 learned_addr learned my addr 172.21.15.6:0/2472201928 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.111+0000 7fa30dd9b640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100c7c00 0x7fa310153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa30e59c640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc0039e0 0x7fa3100c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa30ed9d640 1 -- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100c7c00 msgr2=0x7fa310153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa30ed9d640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100c7c00 0x7fa310153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa30ed9d640 1 -- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc0039e0 msgr2=0x7fa3100c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa30ed9d640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc0039e0 0x7fa3100c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa30ed9d640 1 -- 172.21.15.6:0/2472201928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3101597c0 con 0x7fa3100c5670 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa30ed9d640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3100c5670 0x7fa310163bb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa31015f770 tx=0x7fa30000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa306ffd640 1 -- 172.21.15.6:0/2472201928 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa300019070 con 0x7fa3100c5670 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa306ffd640 1 -- 172.21.15.6:0/2472201928 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa300003990 con 0x7fa3100c5670 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa310159aa0 con 0x7fa3100c5670 2024-03-18T14:43:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa306ffd640 1 -- 172.21.15.6:0/2472201928 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa300003d80 con 0x7fa3100c5670 2024-03-18T14:43:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.112+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3100c5af0 con 0x7fa3100c5670 2024-03-18T14:43:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.113+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31009fa80 con 0x7fa3100c5670 2024-03-18T14:43:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.113+0000 7fa306ffd640 1 -- 172.21.15.6:0/2472201928 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa30001c070 con 0x7fa3100c5670 2024-03-18T14:43:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.114+0000 7fa306ffd640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa2f006bbd0 0x7fa2f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.114+0000 7fa306ffd640 1 -- 172.21.15.6:0/2472201928 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa30008b3f0 con 0x7fa3100c5670 2024-03-18T14:43:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.114+0000 7fa30e59c640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa2f006bbd0 0x7fa2f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.114+0000 7fa30e59c640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa2f006bbd0 0x7fa2f006e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa308034000 tx=0x7fa308009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.118+0000 7fa306ffd640 1 -- 172.21.15.6:0/2472201928 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa300010040 con 0x7fa3100c5670 2024-03-18T14:43:00.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.297+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa31004f620 con 0x7fa2f006bbd0 2024-03-18T14:43:00.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.303+0000 7fa306ffd640 1 -- 172.21.15.6:0/2472201928 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251914 (secure 0 0 0) 0x7fa308082820 con 0x7fa2f006bbd0 2024-03-18T14:43:00.307 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:00.308 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa2f006bbd0 msgr2=0x7fa2f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa2f006bbd0 0x7fa2f006e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa308034000 tx=0x7fa308009030 comp rx=0 tx=0).stop 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3100c5670 msgr2=0x7fa310163bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3100c5670 0x7fa310163bb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa31015f770 tx=0x7fa30000f580 comp rx=0 tx=0).stop 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 shutdown_connections 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa2f006bbd0 0x7fa2f006e090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3100c5670 0x7fa310163bb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3100c7c00 0x7fa310153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 --2- 172.21.15.6:0/2472201928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2fc0039e0 0x7fa3100c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 >> 172.21.15.6:0/2472201928 conn(0x7fa31008b130 msgr2=0x7fa3100cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 shutdown_connections 2024-03-18T14:43:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.306+0000 7fa314cc6640 1 -- 172.21.15.6:0/2472201928 wait complete. 2024-03-18T14:43:00.318 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-03-18T14:42:59.396878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220824,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753896,"statfs":{"total":773094113280,"available":772867989504,"internally_reserved":0,"allocated":4759552,"data_stored":2689676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.100814"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319901+0000","last_change":"2024-03-18T14:42:57.331932+0000","last_active":"2024-03-18T14:42:58.319901+0000","last_peered":"2024-03-18T14:42:58.319901+0000","last_clean":"2024-03-18T14:42:58.319901+0000","last_became_active":"2024-03-18T14:42:57.331648+0000","last_became_peered":"2024-03-18T14:42:57.331648+0000","last_unstale":"2024-03-18T14:42:58.319901+0000","last_undegraded":"2024-03-18T14:42:58.319901+0000","last_fullsized":"2024-03-18T14:42:58.319901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:43.723920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319670+0000","last_change":"2024-03-18T14:42:57.314223+0000","last_active":"2024-03-18T14:42:58.319670+0000","last_peered":"2024-03-18T14:42:58.319670+0000","last_clean":"2024-03-18T14:42:58.319670+0000","last_became_active":"2024-03-18T14:42:57.313928+0000","last_became_peered":"2024-03-18T14:42:57.313928+0000","last_unstale":"2024-03-18T14:42:58.319670+0000","last_undegraded":"2024-03-18T14:42:58.319670+0000","last_fullsized":"2024-03-18T14:42:58.319670+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:34.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334939+0000","last_change":"2024-03-18T14:42:57.326257+0000","last_active":"2024-03-18T14:42:57.334939+0000","last_peered":"2024-03-18T14:42:57.334939+0000","last_clean":"2024-03-18T14:42:57.334939+0000","last_became_active":"2024-03-18T14:42:57.325962+0000","last_became_peered":"2024-03-18T14:42:57.325962+0000","last_unstale":"2024-03-18T14:42:57.334939+0000","last_undegraded":"2024-03-18T14:42:57.334939+0000","last_fullsized":"2024-03-18T14:42:57.334939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330597+0000","last_change":"2024-03-18T14:42:57.323720+0000","last_active":"2024-03-18T14:42:57.330597+0000","last_peered":"2024-03-18T14:42:57.330597+0000","last_clean":"2024-03-18T14:42:57.330597+0000","last_became_active":"2024-03-18T14:42:57.322969+0000","last_became_peered":"2024-03-18T14:42:57.322969+0000","last_unstale":"2024-03-18T14:42:57.330597+0000","last_undegraded":"2024-03-18T14:42:57.330597+0000","last_fullsized":"2024-03-18T14:42:57.330597+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:45.592453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.324807+0000","last_change":"2024-03-18T14:42:57.321988+0000","last_active":"2024-03-18T14:42:57.324807+0000","last_peered":"2024-03-18T14:42:57.324807+0000","last_clean":"2024-03-18T14:42:57.324807+0000","last_became_active":"2024-03-18T14:42:57.321648+0000","last_became_peered":"2024-03-18T14:42:57.321648+0000","last_unstale":"2024-03-18T14:42:57.324807+0000","last_undegraded":"2024-03-18T14:42:57.324807+0000","last_fullsized":"2024-03-18T14:42:57.324807+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:19:34.015756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334767+0000","last_change":"2024-03-18T14:42:57.331614+0000","last_active":"2024-03-18T14:42:57.334767+0000","last_peered":"2024-03-18T14:42:57.334767+0000","last_clean":"2024-03-18T14:42:57.334767+0000","last_became_active":"2024-03-18T14:42:57.331321+0000","last_became_peered":"2024-03-18T14:42:57.331321+0000","last_unstale":"2024-03-18T14:42:57.334767+0000","last_undegraded":"2024-03-18T14:42:57.334767+0000","last_fullsized":"2024-03-18T14:42:57.334767+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324119+0000","last_change":"2024-03-18T14:42:57.324942+0000","last_active":"2024-03-18T14:42:58.324119+0000","last_peered":"2024-03-18T14:42:58.324119+0000","last_clean":"2024-03-18T14:42:58.324119+0000","last_became_active":"2024-03-18T14:42:57.323179+0000","last_became_peered":"2024-03-18T14:42:57.323179+0000","last_unstale":"2024-03-18T14:42:58.324119+0000","last_undegraded":"2024-03-18T14:42:58.324119+0000","last_fullsized":"2024-03-18T14:42:58.324119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:51:26.159733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319426+0000","last_change":"2024-03-18T14:42:57.334380+0000","last_active":"2024-03-18T14:42:58.319426+0000","last_peered":"2024-03-18T14:42:58.319426+0000","last_clean":"2024-03-18T14:42:58.319426+0000","last_became_active":"2024-03-18T14:42:57.334044+0000","last_became_peered":"2024-03-18T14:42:57.334044+0000","last_unstale":"2024-03-18T14:42:58.319426+0000","last_undegraded":"2024-03-18T14:42:58.319426+0000","last_fullsized":"2024-03-18T14:42:58.319426+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:03.049979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.335008+0000","last_change":"2024-03-18T14:42:57.325974+0000","last_active":"2024-03-18T14:42:57.335008+0000","last_peered":"2024-03-18T14:42:57.335008+0000","last_clean":"2024-03-18T14:42:57.335008+0000","last_became_active":"2024-03-18T14:42:57.325631+0000","last_became_peered":"2024-03-18T14:42:57.325631+0000","last_unstale":"2024-03-18T14:42:57.335008+0000","last_undegraded":"2024-03-18T14:42:57.335008+0000","last_fullsized":"2024-03-18T14:42:57.335008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324075+0000","last_change":"2024-03-18T14:42:57.329271+0000","last_active":"2024-03-18T14:42:58.324075+0000","last_peered":"2024-03-18T14:42:58.324075+0000","last_clean":"2024-03-18T14:42:58.324075+0000","last_became_active":"2024-03-18T14:42:57.329010+0000","last_became_peered":"2024-03-18T14:42:57.329010+0000","last_unstale":"2024-03-18T14:42:58.324075+0000","last_undegraded":"2024-03-18T14:42:58.324075+0000","last_fullsized":"2024-03-18T14:42:58.324075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:58.859788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324269+0000","last_change":"2024-03-18T14:42:57.322920+0000","last_active":"2024-03-18T14:42:58.324269+0000","last_peered":"2024-03-18T14:42:58.324269+0000","last_clean":"2024-03-18T14:42:58.324269+0000","last_became_active":"2024-03-18T14:42:57.322640+0000","last_became_peered":"2024-03-18T14:42:57.322640+0000","last_unstale":"2024-03-18T14:42:58.324269+0000","last_undegraded":"2024-03-18T14:42:58.324269+0000","last_fullsized":"2024-03-18T14:42:58.324269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:36.168961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334989+0000","last_change":"2024-03-18T14:42:57.328727+0000","last_active":"2024-03-18T14:42:57.334989+0000","last_peered":"2024-03-18T14:42:57.334989+0000","last_clean":"2024-03-18T14:42:57.334989+0000","last_became_active":"2024-03-18T14:42:57.328372+0000","last_became_peered":"2024-03-18T14:42:57.328372+0000","last_unstale":"2024-03-18T14:42:57.334989+0000","last_undegraded":"2024-03-18T14:42:57.334989+0000","last_fullsized":"2024-03-18T14:42:57.334989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324057+0000","last_change":"2024-03-18T14:42:57.331522+0000","last_active":"2024-03-18T14:42:58.324057+0000","last_peered":"2024-03-18T14:42:58.324057+0000","last_clean":"2024-03-18T14:42:58.324057+0000","last_became_active":"2024-03-18T14:42:57.329664+0000","last_became_peered":"2024-03-18T14:42:57.329664+0000","last_unstale":"2024-03-18T14:42:58.324057+0000","last_undegraded":"2024-03-18T14:42:58.324057+0000","last_fullsized":"2024-03-18T14:42:58.324057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:51:04.081506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333450+0000","last_change":"2024-03-18T14:42:57.329322+0000","last_active":"2024-03-18T14:42:57.333450+0000","last_peered":"2024-03-18T14:42:57.333450+0000","last_clean":"2024-03-18T14:42:57.333450+0000","last_became_active":"2024-03-18T14:42:57.329033+0000","last_became_peered":"2024-03-18T14:42:57.329033+0000","last_unstale":"2024-03-18T14:42:57.333450+0000","last_undegraded":"2024-03-18T14:42:57.333450+0000","last_fullsized":"2024-03-18T14:42:57.333450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:36.914241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324154+0000","last_change":"2024-03-18T14:42:57.324633+0000","last_active":"2024-03-18T14:42:58.324154+0000","last_peered":"2024-03-18T14:42:58.324154+0000","last_clean":"2024-03-18T14:42:58.324154+0000","last_became_active":"2024-03-18T14:42:57.322820+0000","last_became_peered":"2024-03-18T14:42:57.322820+0000","last_unstale":"2024-03-18T14:42:58.324154+0000","last_undegraded":"2024-03-18T14:42:58.324154+0000","last_fullsized":"2024-03-18T14:42:58.324154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:33:17.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.326682+0000","last_change":"2024-03-18T14:42:57.320474+0000","last_active":"2024-03-18T14:42:57.326682+0000","last_peered":"2024-03-18T14:42:57.326682+0000","last_clean":"2024-03-18T14:42:57.326682+0000","last_became_active":"2024-03-18T14:42:57.320076+0000","last_became_peered":"2024-03-18T14:42:57.320076+0000","last_unstale":"2024-03-18T14:42:57.326682+0000","last_undegraded":"2024-03-18T14:42:57.326682+0000","last_fullsized":"2024-03-18T14:42:57.326682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:19.966646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.327211+0000","last_change":"2024-03-18T14:42:57.323775+0000","last_active":"2024-03-18T14:42:57.327211+0000","last_peered":"2024-03-18T14:42:57.327211+0000","last_clean":"2024-03-18T14:42:57.327211+0000","last_became_active":"2024-03-18T14:42:57.323496+0000","last_became_peered":"2024-03-18T14:42:57.323496+0000","last_unstale":"2024-03-18T14:42:57.327211+0000","last_undegraded":"2024-03-18T14:42:57.327211+0000","last_fullsized":"2024-03-18T14:42:57.327211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:29.356561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323602+0000","last_change":"2024-03-18T14:42:57.328474+0000","last_active":"2024-03-18T14:42:58.323602+0000","last_peered":"2024-03-18T14:42:58.323602+0000","last_clean":"2024-03-18T14:42:58.323602+0000","last_became_active":"2024-03-18T14:42:57.328224+0000","last_became_peered":"2024-03-18T14:42:57.328224+0000","last_unstale":"2024-03-18T14:42:58.323602+0000","last_undegraded":"2024-03-18T14:42:58.323602+0000","last_fullsized":"2024-03-18T14:42:58.323602+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330433+0000","last_change":"2024-03-18T14:42:57.323110+0000","last_active":"2024-03-18T14:42:57.330433+0000","last_peered":"2024-03-18T14:42:57.330433+0000","last_clean":"2024-03-18T14:42:57.330433+0000","last_became_active":"2024-03-18T14:42:57.322800+0000","last_became_peered":"2024-03-18T14:42:57.322800+0000","last_unstale":"2024-03-18T14:42:57.330433+0000","last_undegraded":"2024-03-18T14:42:57.330433+0000","last_fullsized":"2024-03-18T14:42:57.330433+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:21:21.131435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323194+0000","last_change":"2024-03-18T14:42:57.318916+0000","last_active":"2024-03-18T14:42:58.323194+0000","last_peered":"2024-03-18T14:42:58.323194+0000","last_clean":"2024-03-18T14:42:58.323194+0000","last_became_active":"2024-03-18T14:42:57.318351+0000","last_became_peered":"2024-03-18T14:42:57.318351+0000","last_unstale":"2024-03-18T14:42:58.323194+0000","last_undegraded":"2024-03-18T14:42:58.323194+0000","last_fullsized":"2024-03-18T14:42:58.323194+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:13.142420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324145+0000","last_change":"2024-03-18T14:42:57.322170+0000","last_active":"2024-03-18T14:42:58.324145+0000","last_peered":"2024-03-18T14:42:58.324145+0000","last_clean":"2024-03-18T14:42:58.324145+0000","last_became_active":"2024-03-18T14:42:57.321891+0000","last_became_peered":"2024-03-18T14:42:57.321891+0000","last_unstale":"2024-03-18T14:42:58.324145+0000","last_undegraded":"2024-03-18T14:42:58.324145+0000","last_fullsized":"2024-03-18T14:42:58.324145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:22.016912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333247+0000","last_change":"2024-03-18T14:42:57.325931+0000","last_active":"2024-03-18T14:42:57.333247+0000","last_peered":"2024-03-18T14:42:57.333247+0000","last_clean":"2024-03-18T14:42:57.333247+0000","last_became_active":"2024-03-18T14:42:57.325639+0000","last_became_peered":"2024-03-18T14:42:57.325639+0000","last_unstale":"2024-03-18T14:42:57.333247+0000","last_undegraded":"2024-03-18T14:42:57.333247+0000","last_fullsized":"2024-03-18T14:42:57.333247+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.325572+0000","last_change":"2024-03-18T14:42:57.316559+0000","last_active":"2024-03-18T14:42:57.325572+0000","last_peered":"2024-03-18T14:42:57.325572+0000","last_clean":"2024-03-18T14:42:57.325572+0000","last_became_active":"2024-03-18T14:42:57.315949+0000","last_became_peered":"2024-03-18T14:42:57.315949+0000","last_unstale":"2024-03-18T14:42:57.325572+0000","last_undegraded":"2024-03-18T14:42:57.325572+0000","last_fullsized":"2024-03-18T14:42:57.325572+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.314576+0000","last_change":"2024-03-18T14:42:57.312017+0000","last_active":"2024-03-18T14:42:57.314576+0000","last_peered":"2024-03-18T14:42:57.314576+0000","last_clean":"2024-03-18T14:42:57.314576+0000","last_became_active":"2024-03-18T14:42:57.311706+0000","last_became_peered":"2024-03-18T14:42:57.311706+0000","last_unstale":"2024-03-18T14:42:57.314576+0000","last_undegraded":"2024-03-18T14:42:57.314576+0000","last_fullsized":"2024-03-18T14:42:57.314576+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:05.687491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323967+0000","last_change":"2024-03-18T14:42:57.324082+0000","last_active":"2024-03-18T14:42:58.323967+0000","last_peered":"2024-03-18T14:42:58.323967+0000","last_clean":"2024-03-18T14:42:58.323967+0000","last_became_active":"2024-03-18T14:42:57.323801+0000","last_became_peered":"2024-03-18T14:42:57.323801+0000","last_unstale":"2024-03-18T14:42:58.323967+0000","last_undegraded":"2024-03-18T14:42:58.323967+0000","last_fullsized":"2024-03-18T14:42:58.323967+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330467+0000","last_change":"2024-03-18T14:42:57.327616+0000","last_active":"2024-03-18T14:42:57.330467+0000","last_peered":"2024-03-18T14:42:57.330467+0000","last_clean":"2024-03-18T14:42:57.330467+0000","last_became_active":"2024-03-18T14:42:57.327327+0000","last_became_peered":"2024-03-18T14:42:57.327327+0000","last_unstale":"2024-03-18T14:42:57.330467+0000","last_undegraded":"2024-03-18T14:42:57.330467+0000","last_fullsized":"2024-03-18T14:42:57.330467+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323346+0000","last_change":"2024-03-18T14:42:57.313584+0000","last_active":"2024-03-18T14:42:58.323346+0000","last_peered":"2024-03-18T14:42:58.323346+0000","last_clean":"2024-03-18T14:42:58.323346+0000","last_became_active":"2024-03-18T14:42:57.313306+0000","last_became_peered":"2024-03-18T14:42:57.313306+0000","last_unstale":"2024-03-18T14:42:58.323346+0000","last_undegraded":"2024-03-18T14:42:58.323346+0000","last_fullsized":"2024-03-18T14:42:58.323346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:19.877972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324190+0000","last_change":"2024-03-18T14:42:57.321881+0000","last_active":"2024-03-18T14:42:58.324190+0000","last_peered":"2024-03-18T14:42:58.324190+0000","last_clean":"2024-03-18T14:42:58.324190+0000","last_became_active":"2024-03-18T14:42:57.321599+0000","last_became_peered":"2024-03-18T14:42:57.321599+0000","last_unstale":"2024-03-18T14:42:58.324190+0000","last_undegraded":"2024-03-18T14:42:58.324190+0000","last_fullsized":"2024-03-18T14:42:58.324190+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:11.493713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323437+0000","last_change":"2024-03-18T14:42:57.318718+0000","last_active":"2024-03-18T14:42:58.323437+0000","last_peered":"2024-03-18T14:42:58.323437+0000","last_clean":"2024-03-18T14:42:58.323437+0000","last_became_active":"2024-03-18T14:42:57.318427+0000","last_became_peered":"2024-03-18T14:42:57.318427+0000","last_unstale":"2024-03-18T14:42:58.323437+0000","last_undegraded":"2024-03-18T14:42:58.323437+0000","last_fullsized":"2024-03-18T14:42:58.323437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323725+0000","last_change":"2024-03-18T14:42:57.314465+0000","last_active":"2024-03-18T14:42:58.323725+0000","last_peered":"2024-03-18T14:42:58.323725+0000","last_clean":"2024-03-18T14:42:58.323725+0000","last_became_active":"2024-03-18T14:42:57.314223+0000","last_became_peered":"2024-03-18T14:42:57.314223+0000","last_unstale":"2024-03-18T14:42:58.323725+0000","last_undegraded":"2024-03-18T14:42:58.323725+0000","last_fullsized":"2024-03-18T14:42:58.323725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.325553+0000","last_change":"2024-03-18T14:42:57.316110+0000","last_active":"2024-03-18T14:42:57.325553+0000","last_peered":"2024-03-18T14:42:57.325553+0000","last_clean":"2024-03-18T14:42:57.325553+0000","last_became_active":"2024-03-18T14:42:57.315831+0000","last_became_peered":"2024-03-18T14:42:57.315831+0000","last_unstale":"2024-03-18T14:42:57.325553+0000","last_undegraded":"2024-03-18T14:42:57.325553+0000","last_fullsized":"2024-03-18T14:42:57.325553+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334640+0000","last_change":"2024-03-18T14:42:57.331604+0000","last_active":"2024-03-18T14:42:57.334640+0000","last_peered":"2024-03-18T14:42:57.334640+0000","last_clean":"2024-03-18T14:42:57.334640+0000","last_became_active":"2024-03-18T14:42:57.331283+0000","last_became_peered":"2024-03-18T14:42:57.331283+0000","last_unstale":"2024-03-18T14:42:57.334640+0000","last_undegraded":"2024-03-18T14:42:57.334640+0000","last_fullsized":"2024-03-18T14:42:57.334640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.327119+0000","last_change":"2024-03-18T14:42:57.325069+0000","last_active":"2024-03-18T14:42:57.327119+0000","last_peered":"2024-03-18T14:42:57.327119+0000","last_clean":"2024-03-18T14:42:57.327119+0000","last_became_active":"2024-03-18T14:42:57.324466+0000","last_became_peered":"2024-03-18T14:42:57.324466+0000","last_unstale":"2024-03-18T14:42:57.327119+0000","last_undegraded":"2024-03-18T14:42:57.327119+0000","last_fullsized":"2024-03-18T14:42:57.327119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:07.247126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323957+0000","last_change":"2024-03-18T14:42:56.316599+0000","last_active":"2024-03-18T14:42:58.323957+0000","last_peered":"2024-03-18T14:42:58.323957+0000","last_clean":"2024-03-18T14:42:58.323957+0000","last_became_active":"2024-03-18T14:42:56.316212+0000","last_became_peered":"2024-03-18T14:42:56.316212+0000","last_unstale":"2024-03-18T14:42:58.323957+0000","last_undegraded":"2024-03-18T14:42:58.323957+0000","last_fullsized":"2024-03-18T14:42:58.323957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:04:45.807569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330528+0000","last_change":"2024-03-18T14:42:57.326929+0000","last_active":"2024-03-18T14:42:57.330528+0000","last_peered":"2024-03-18T14:42:57.330528+0000","last_clean":"2024-03-18T14:42:57.330528+0000","last_became_active":"2024-03-18T14:42:57.326648+0000","last_became_peered":"2024-03-18T14:42:57.326648+0000","last_unstale":"2024-03-18T14:42:57.330528+0000","last_undegraded":"2024-03-18T14:42:57.330528+0000","last_fullsized":"2024-03-18T14:42:57.330528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:19.877972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.329215+0000","last_change":"2024-03-18T14:42:56.324129+0000","last_active":"2024-03-18T14:42:57.329215+0000","last_peered":"2024-03-18T14:42:57.329215+0000","last_clean":"2024-03-18T14:42:57.329215+0000","last_became_active":"2024-03-18T14:42:56.323848+0000","last_became_peered":"2024-03-18T14:42:56.323848+0000","last_unstale":"2024-03-18T14:42:57.329215+0000","last_undegraded":"2024-03-18T14:42:57.329215+0000","last_fullsized":"2024-03-18T14:42:57.329215+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:09.445509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333709+0000","last_change":"2024-03-18T14:42:57.331045+0000","last_active":"2024-03-18T14:42:57.333709+0000","last_peered":"2024-03-18T14:42:57.333709+0000","last_clean":"2024-03-18T14:42:57.333709+0000","last_became_active":"2024-03-18T14:42:57.330772+0000","last_became_peered":"2024-03-18T14:42:57.330772+0000","last_unstale":"2024-03-18T14:42:57.333709+0000","last_undegraded":"2024-03-18T14:42:57.333709+0000","last_fullsized":"2024-03-18T14:42:57.333709+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:37.127269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333492+0000","last_change":"2024-03-18T14:42:56.318507+0000","last_active":"2024-03-18T14:42:57.333492+0000","last_peered":"2024-03-18T14:42:57.333492+0000","last_clean":"2024-03-18T14:42:57.333492+0000","last_became_active":"2024-03-18T14:42:56.317461+0000","last_became_peered":"2024-03-18T14:42:56.317461+0000","last_unstale":"2024-03-18T14:42:57.333492+0000","last_undegraded":"2024-03-18T14:42:57.333492+0000","last_fullsized":"2024-03-18T14:42:57.333492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:51.796731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319858+0000","last_change":"2024-03-18T14:42:57.331698+0000","last_active":"2024-03-18T14:42:58.319858+0000","last_peered":"2024-03-18T14:42:58.319858+0000","last_clean":"2024-03-18T14:42:58.319858+0000","last_became_active":"2024-03-18T14:42:57.331247+0000","last_became_peered":"2024-03-18T14:42:57.331247+0000","last_unstale":"2024-03-18T14:42:58.319858+0000","last_undegraded":"2024-03-18T14:42:58.319858+0000","last_fullsized":"2024-03-18T14:42:58.319858+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:58.272509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.336687+0000","last_change":"2024-03-18T14:42:56.319748+0000","last_active":"2024-03-18T14:42:57.336687+0000","last_peered":"2024-03-18T14:42:57.336687+0000","last_clean":"2024-03-18T14:42:57.336687+0000","last_became_active":"2024-03-18T14:42:56.319544+0000","last_became_peered":"2024-03-18T14:42:56.319544+0000","last_unstale":"2024-03-18T14:42:57.336687+0000","last_undegraded":"2024-03-18T14:42:57.336687+0000","last_fullsized":"2024-03-18T14:42:57.336687+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:12.291125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334829+0000","last_change":"2024-03-18T14:42:57.325466+0000","last_active":"2024-03-18T14:42:57.334829+0000","last_peered":"2024-03-18T14:42:57.334829+0000","last_clean":"2024-03-18T14:42:57.334829+0000","last_became_active":"2024-03-18T14:42:57.325123+0000","last_became_peered":"2024-03-18T14:42:57.325123+0000","last_unstale":"2024-03-18T14:42:57.334829+0000","last_undegraded":"2024-03-18T14:42:57.334829+0000","last_fullsized":"2024-03-18T14:42:57.334829+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:13.142420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324339+0000","last_change":"2024-03-18T14:42:56.318873+0000","last_active":"2024-03-18T14:42:58.324339+0000","last_peered":"2024-03-18T14:42:58.324339+0000","last_clean":"2024-03-18T14:42:58.324339+0000","last_became_active":"2024-03-18T14:42:56.318585+0000","last_became_peered":"2024-03-18T14:42:56.318585+0000","last_unstale":"2024-03-18T14:42:58.324339+0000","last_undegraded":"2024-03-18T14:42:58.324339+0000","last_fullsized":"2024-03-18T14:42:58.324339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:25.894064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324320+0000","last_change":"2024-03-18T14:42:57.325189+0000","last_active":"2024-03-18T14:42:58.324320+0000","last_peered":"2024-03-18T14:42:58.324320+0000","last_clean":"2024-03-18T14:42:58.324320+0000","last_became_active":"2024-03-18T14:42:57.324545+0000","last_became_peered":"2024-03-18T14:42:57.324545+0000","last_unstale":"2024-03-18T14:42:58.324320+0000","last_undegraded":"2024-03-18T14:42:58.324320+0000","last_fullsized":"2024-03-18T14:42:58.324320+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:34.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.336457+0000","last_change":"2024-03-18T14:42:56.323154+0000","last_active":"2024-03-18T14:42:57.336457+0000","last_peered":"2024-03-18T14:42:57.336457+0000","last_clean":"2024-03-18T14:42:57.336457+0000","last_became_active":"2024-03-18T14:42:56.322880+0000","last_became_peered":"2024-03-18T14:42:56.322880+0000","last_unstale":"2024-03-18T14:42:57.336457+0000","last_undegraded":"2024-03-18T14:42:57.336457+0000","last_fullsized":"2024-03-18T14:42:57.336457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:18.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332979+0000","last_change":"2024-03-18T14:42:57.322920+0000","last_active":"2024-03-18T14:42:57.332979+0000","last_peered":"2024-03-18T14:42:57.332979+0000","last_clean":"2024-03-18T14:42:57.332979+0000","last_became_active":"2024-03-18T14:42:57.322655+0000","last_became_peered":"2024-03-18T14:42:57.322655+0000","last_unstale":"2024-03-18T14:42:57.332979+0000","last_undegraded":"2024-03-18T14:42:57.332979+0000","last_fullsized":"2024-03-18T14:42:57.332979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.312580+0000","last_change":"2024-03-18T14:42:56.317870+0000","last_active":"2024-03-18T14:42:57.312580+0000","last_peered":"2024-03-18T14:42:57.312580+0000","last_clean":"2024-03-18T14:42:57.312580+0000","last_became_active":"2024-03-18T14:42:56.317610+0000","last_became_peered":"2024-03-18T14:42:56.317610+0000","last_unstale":"2024-03-18T14:42:57.312580+0000","last_undegraded":"2024-03-18T14:42:57.312580+0000","last_fullsized":"2024-03-18T14:42:57.312580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:09.445509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319446+0000","last_change":"2024-03-18T14:42:57.328371+0000","last_active":"2024-03-18T14:42:58.319446+0000","last_peered":"2024-03-18T14:42:58.319446+0000","last_clean":"2024-03-18T14:42:58.319446+0000","last_became_active":"2024-03-18T14:42:57.328065+0000","last_became_peered":"2024-03-18T14:42:57.328065+0000","last_unstale":"2024-03-18T14:42:58.319446+0000","last_undegraded":"2024-03-18T14:42:58.319446+0000","last_fullsized":"2024-03-18T14:42:58.319446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:22.016912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.313104+0000","last_change":"2024-03-18T14:42:56.316817+0000","last_active":"2024-03-18T14:42:57.313104+0000","last_peered":"2024-03-18T14:42:57.313104+0000","last_clean":"2024-03-18T14:42:57.313104+0000","last_became_active":"2024-03-18T14:42:56.316547+0000","last_became_peered":"2024-03-18T14:42:56.316547+0000","last_unstale":"2024-03-18T14:42:57.313104+0000","last_undegraded":"2024-03-18T14:42:57.313104+0000","last_fullsized":"2024-03-18T14:42:57.313104+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:51.796731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330575+0000","last_change":"2024-03-18T14:42:57.323438+0000","last_active":"2024-03-18T14:42:57.330575+0000","last_peered":"2024-03-18T14:42:57.330575+0000","last_clean":"2024-03-18T14:42:57.330575+0000","last_became_active":"2024-03-18T14:42:57.322586+0000","last_became_peered":"2024-03-18T14:42:57.322586+0000","last_unstale":"2024-03-18T14:42:57.330575+0000","last_undegraded":"2024-03-18T14:42:57.330575+0000","last_fullsized":"2024-03-18T14:42:57.330575+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:04:46.809699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319907+0000","last_change":"2024-03-18T14:42:56.318787+0000","last_active":"2024-03-18T14:42:58.319907+0000","last_peered":"2024-03-18T14:42:58.319907+0000","last_clean":"2024-03-18T14:42:58.319907+0000","last_became_active":"2024-03-18T14:42:56.318483+0000","last_became_peered":"2024-03-18T14:42:56.318483+0000","last_unstale":"2024-03-18T14:42:58.319907+0000","last_undegraded":"2024-03-18T14:42:58.319907+0000","last_fullsized":"2024-03-18T14:42:58.319907+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.244996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.329040+0000","last_change":"2024-03-18T14:42:57.317184+0000","last_active":"2024-03-18T14:42:57.329040+0000","last_peered":"2024-03-18T14:42:57.329040+0000","last_clean":"2024-03-18T14:42:57.329040+0000","last_became_active":"2024-03-18T14:42:57.316925+0000","last_became_peered":"2024-03-18T14:42:57.316925+0000","last_unstale":"2024-03-18T14:42:57.329040+0000","last_undegraded":"2024-03-18T14:42:57.329040+0000","last_fullsized":"2024-03-18T14:42:57.329040+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:36.914241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323390+0000","last_change":"2024-03-18T14:42:56.321703+0000","last_active":"2024-03-18T14:42:58.323390+0000","last_peered":"2024-03-18T14:42:58.323390+0000","last_clean":"2024-03-18T14:42:58.323390+0000","last_became_active":"2024-03-18T14:42:56.321495+0000","last_became_peered":"2024-03-18T14:42:56.321495+0000","last_unstale":"2024-03-18T14:42:58.323390+0000","last_undegraded":"2024-03-18T14:42:58.323390+0000","last_fullsized":"2024-03-18T14:42:58.323390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:14.308814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334978+0000","last_change":"2024-03-18T14:42:57.329512+0000","last_active":"2024-03-18T14:42:57.334978+0000","last_peered":"2024-03-18T14:42:57.334978+0000","last_clean":"2024-03-18T14:42:57.334978+0000","last_became_active":"2024-03-18T14:42:57.328808+0000","last_became_peered":"2024-03-18T14:42:57.328808+0000","last_unstale":"2024-03-18T14:42:57.334978+0000","last_undegraded":"2024-03-18T14:42:57.334978+0000","last_fullsized":"2024-03-18T14:42:57.334978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324370+0000","last_change":"2024-03-18T14:42:56.315722+0000","last_active":"2024-03-18T14:42:58.324370+0000","last_peered":"2024-03-18T14:42:58.324370+0000","last_clean":"2024-03-18T14:42:58.324370+0000","last_became_active":"2024-03-18T14:42:56.315354+0000","last_became_peered":"2024-03-18T14:42:56.315354+0000","last_unstale":"2024-03-18T14:42:58.324370+0000","last_undegraded":"2024-03-18T14:42:58.324370+0000","last_fullsized":"2024-03-18T14:42:58.324370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:54.845213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333162+0000","last_change":"2024-03-18T14:42:57.325238+0000","last_active":"2024-03-18T14:42:57.333162+0000","last_peered":"2024-03-18T14:42:57.333162+0000","last_clean":"2024-03-18T14:42:57.333162+0000","last_became_active":"2024-03-18T14:42:57.324991+0000","last_became_peered":"2024-03-18T14:42:57.324991+0000","last_unstale":"2024-03-18T14:42:57.333162+0000","last_undegraded":"2024-03-18T14:42:57.333162+0000","last_fullsized":"2024-03-18T14:42:57.333162+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319762+0000","last_change":"2024-03-18T14:42:56.323408+0000","last_active":"2024-03-18T14:42:58.319762+0000","last_peered":"2024-03-18T14:42:58.319762+0000","last_clean":"2024-03-18T14:42:58.319762+0000","last_became_active":"2024-03-18T14:42:56.323126+0000","last_became_peered":"2024-03-18T14:42:56.323126+0000","last_unstale":"2024-03-18T14:42:58.319762+0000","last_undegraded":"2024-03-18T14:42:58.319762+0000","last_fullsized":"2024-03-18T14:42:58.319762+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:10.938463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334857+0000","last_change":"2024-03-18T14:42:57.325813+0000","last_active":"2024-03-18T14:42:57.334857+0000","last_peered":"2024-03-18T14:42:57.334857+0000","last_clean":"2024-03-18T14:42:57.334857+0000","last_became_active":"2024-03-18T14:42:57.325517+0000","last_became_peered":"2024-03-18T14:42:57.325517+0000","last_unstale":"2024-03-18T14:42:57.334857+0000","last_undegraded":"2024-03-18T14:42:57.334857+0000","last_fullsized":"2024-03-18T14:42:57.334857+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:19.877972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319697+0000","last_change":"2024-03-18T14:42:56.321185+0000","last_active":"2024-03-18T14:42:58.319697+0000","last_peered":"2024-03-18T14:42:58.319697+0000","last_clean":"2024-03-18T14:42:58.319697+0000","last_became_active":"2024-03-18T14:42:56.320895+0000","last_became_peered":"2024-03-18T14:42:56.320895+0000","last_unstale":"2024-03-18T14:42:58.319697+0000","last_undegraded":"2024-03-18T14:42:58.319697+0000","last_fullsized":"2024-03-18T14:42:58.319697+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:18.964516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319471+0000","last_change":"2024-03-18T14:42:57.333088+0000","last_active":"2024-03-18T14:42:58.319471+0000","last_peered":"2024-03-18T14:42:58.319471+0000","last_clean":"2024-03-18T14:42:58.319471+0000","last_became_active":"2024-03-18T14:42:57.332789+0000","last_became_peered":"2024-03-18T14:42:57.332789+0000","last_unstale":"2024-03-18T14:42:58.319471+0000","last_undegraded":"2024-03-18T14:42:58.319471+0000","last_fullsized":"2024-03-18T14:42:58.319471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:33:17.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.315285+0000","last_change":"2024-03-18T14:42:56.318459+0000","last_active":"2024-03-18T14:42:57.315285+0000","last_peered":"2024-03-18T14:42:57.315285+0000","last_clean":"2024-03-18T14:42:57.315285+0000","last_became_active":"2024-03-18T14:42:56.318100+0000","last_became_peered":"2024-03-18T14:42:56.318100+0000","last_unstale":"2024-03-18T14:42:57.315285+0000","last_undegraded":"2024-03-18T14:42:57.315285+0000","last_fullsized":"2024-03-18T14:42:57.315285+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:12.140291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.322115+0000","last_change":"2024-03-18T14:42:57.313551+0000","last_active":"2024-03-18T14:42:57.322115+0000","last_peered":"2024-03-18T14:42:57.322115+0000","last_clean":"2024-03-18T14:42:57.322115+0000","last_became_active":"2024-03-18T14:42:57.313303+0000","last_became_peered":"2024-03-18T14:42:57.313303+0000","last_unstale":"2024-03-18T14:42:57.322115+0000","last_undegraded":"2024-03-18T14:42:57.322115+0000","last_fullsized":"2024-03-18T14:42:57.322115+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.322980+0000","last_change":"2024-03-18T14:42:56.317018+0000","last_active":"2024-03-18T14:42:58.322980+0000","last_peered":"2024-03-18T14:42:58.322980+0000","last_clean":"2024-03-18T14:42:58.322980+0000","last_became_active":"2024-03-18T14:42:56.316710+0000","last_became_peered":"2024-03-18T14:42:56.316710+0000","last_unstale":"2024-03-18T14:42:58.322980+0000","last_undegraded":"2024-03-18T14:42:58.322980+0000","last_fullsized":"2024-03-18T14:42:58.322980+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:44.590323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319596+0000","last_change":"2024-03-18T14:42:57.332643+0000","last_active":"2024-03-18T14:42:58.319596+0000","last_peered":"2024-03-18T14:42:58.319596+0000","last_clean":"2024-03-18T14:42:58.319596+0000","last_became_active":"2024-03-18T14:42:57.332353+0000","last_became_peered":"2024-03-18T14:42:57.332353+0000","last_unstale":"2024-03-18T14:42:58.319596+0000","last_undegraded":"2024-03-18T14:42:58.319596+0000","last_fullsized":"2024-03-18T14:42:58.319596+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:11.493713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.339419+0000","last_change":"2024-03-18T14:42:56.323694+0000","last_active":"2024-03-18T14:42:57.339419+0000","last_peered":"2024-03-18T14:42:57.339419+0000","last_clean":"2024-03-18T14:42:57.339419+0000","last_became_active":"2024-03-18T14:42:56.323399+0000","last_became_peered":"2024-03-18T14:42:56.323399+0000","last_unstale":"2024-03-18T14:42:57.339419+0000","last_undegraded":"2024-03-18T14:42:57.339419+0000","last_fullsized":"2024-03-18T14:42:57.339419+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:12.140291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319745+0000","last_change":"2024-03-18T14:42:57.328096+0000","last_active":"2024-03-18T14:42:58.319745+0000","last_peered":"2024-03-18T14:42:58.319745+0000","last_clean":"2024-03-18T14:42:58.319745+0000","last_became_active":"2024-03-18T14:42:57.327794+0000","last_became_peered":"2024-03-18T14:42:57.327794+0000","last_unstale":"2024-03-18T14:42:58.319745+0000","last_undegraded":"2024-03-18T14:42:58.319745+0000","last_fullsized":"2024-03-18T14:42:58.319745+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:53.015812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323948+0000","last_change":"2024-03-18T14:42:56.313601+0000","last_active":"2024-03-18T14:42:58.323948+0000","last_peered":"2024-03-18T14:42:58.323948+0000","last_clean":"2024-03-18T14:42:58.323948+0000","last_became_active":"2024-03-18T14:42:56.313307+0000","last_became_peered":"2024-03-18T14:42:56.313307+0000","last_unstale":"2024-03-18T14:42:58.323948+0000","last_undegraded":"2024-03-18T14:42:58.323948+0000","last_fullsized":"2024-03-18T14:42:58.323948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:35.912112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324487+0000","last_change":"2024-03-18T14:42:57.319047+0000","last_active":"2024-03-18T14:42:58.324487+0000","last_peered":"2024-03-18T14:42:58.324487+0000","last_clean":"2024-03-18T14:42:58.324487+0000","last_became_active":"2024-03-18T14:42:57.318766+0000","last_became_peered":"2024-03-18T14:42:57.318766+0000","last_unstale":"2024-03-18T14:42:58.324487+0000","last_undegraded":"2024-03-18T14:42:58.324487+0000","last_fullsized":"2024-03-18T14:42:58.324487+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:07.997004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333654+0000","last_change":"2024-03-18T14:42:56.319098+0000","last_active":"2024-03-18T14:42:57.333654+0000","last_peered":"2024-03-18T14:42:57.333654+0000","last_clean":"2024-03-18T14:42:57.333654+0000","last_became_active":"2024-03-18T14:42:56.318482+0000","last_became_peered":"2024-03-18T14:42:56.318482+0000","last_unstale":"2024-03-18T14:42:57.333654+0000","last_undegraded":"2024-03-18T14:42:57.333654+0000","last_fullsized":"2024-03-18T14:42:57.333654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:18.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332872+0000","last_change":"2024-03-18T14:42:57.319520+0000","last_active":"2024-03-18T14:42:57.332872+0000","last_peered":"2024-03-18T14:42:57.332872+0000","last_clean":"2024-03-18T14:42:57.332872+0000","last_became_active":"2024-03-18T14:42:57.319194+0000","last_became_peered":"2024-03-18T14:42:57.319194+0000","last_unstale":"2024-03-18T14:42:57.332872+0000","last_undegraded":"2024-03-18T14:42:57.332872+0000","last_fullsized":"2024-03-18T14:42:57.332872+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333350+0000","last_change":"2024-03-18T14:42:56.320173+0000","last_active":"2024-03-18T14:42:57.333350+0000","last_peered":"2024-03-18T14:42:57.333350+0000","last_clean":"2024-03-18T14:42:57.333350+0000","last_became_active":"2024-03-18T14:42:56.319845+0000","last_became_peered":"2024-03-18T14:42:56.319845+0000","last_unstale":"2024-03-18T14:42:57.333350+0000","last_undegraded":"2024-03-18T14:42:57.333350+0000","last_fullsized":"2024-03-18T14:42:57.333350+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:10:30.596234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.323490+0000","last_change":"2024-03-18T14:42:57.320961+0000","last_active":"2024-03-18T14:42:57.323490+0000","last_peered":"2024-03-18T14:42:57.323490+0000","last_clean":"2024-03-18T14:42:57.323490+0000","last_became_active":"2024-03-18T14:42:57.320679+0000","last_became_peered":"2024-03-18T14:42:57.320679+0000","last_unstale":"2024-03-18T14:42:57.323490+0000","last_undegraded":"2024-03-18T14:42:57.323490+0000","last_fullsized":"2024-03-18T14:42:57.323490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:22.091349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.313155+0000","last_change":"2024-03-18T14:42:56.317130+0000","last_active":"2024-03-18T14:42:57.313155+0000","last_peered":"2024-03-18T14:42:57.313155+0000","last_clean":"2024-03-18T14:42:57.313155+0000","last_became_active":"2024-03-18T14:42:56.316836+0000","last_became_peered":"2024-03-18T14:42:56.316836+0000","last_unstale":"2024-03-18T14:42:57.313155+0000","last_undegraded":"2024-03-18T14:42:57.313155+0000","last_fullsized":"2024-03-18T14:42:57.313155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:12.291125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.326415+0000","last_change":"2024-03-18T14:42:57.317216+0000","last_active":"2024-03-18T14:42:57.326415+0000","last_peered":"2024-03-18T14:42:57.326415+0000","last_clean":"2024-03-18T14:42:57.326415+0000","last_became_active":"2024-03-18T14:42:57.316940+0000","last_became_peered":"2024-03-18T14:42:57.316940+0000","last_unstale":"2024-03-18T14:42:57.326415+0000","last_undegraded":"2024-03-18T14:42:57.326415+0000","last_fullsized":"2024-03-18T14:42:57.326415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:37.127269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.335432+0000","last_change":"2024-03-18T14:42:56.320581+0000","last_active":"2024-03-18T14:42:57.335432+0000","last_peered":"2024-03-18T14:42:57.335432+0000","last_clean":"2024-03-18T14:42:57.335432+0000","last_became_active":"2024-03-18T14:42:56.319720+0000","last_became_peered":"2024-03-18T14:42:56.319720+0000","last_unstale":"2024-03-18T14:42:57.335432+0000","last_undegraded":"2024-03-18T14:42:57.335432+0000","last_fullsized":"2024-03-18T14:42:57.335432+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:03.169629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334998+0000","last_change":"2024-03-18T14:42:57.329810+0000","last_active":"2024-03-18T14:42:57.334998+0000","last_peered":"2024-03-18T14:42:57.334998+0000","last_clean":"2024-03-18T14:42:57.334998+0000","last_became_active":"2024-03-18T14:42:57.329517+0000","last_became_peered":"2024-03-18T14:42:57.329517+0000","last_unstale":"2024-03-18T14:42:57.334998+0000","last_undegraded":"2024-03-18T14:42:57.334998+0000","last_fullsized":"2024-03-18T14:42:57.334998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:36.914241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.305706+0000","last_change":"2024-03-18T14:42:56.314280+0000","last_active":"2024-03-18T14:42:57.305706+0000","last_peered":"2024-03-18T14:42:57.305706+0000","last_clean":"2024-03-18T14:42:57.305706+0000","last_became_active":"2024-03-18T14:42:56.313977+0000","last_became_peered":"2024-03-18T14:42:56.313977+0000","last_unstale":"2024-03-18T14:42:57.305706+0000","last_undegraded":"2024-03-18T14:42:57.305706+0000","last_fullsized":"2024-03-18T14:42:57.305706+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:55.790619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324123+0000","last_change":"2024-03-18T14:42:57.327062+0000","last_active":"2024-03-18T14:42:58.324123+0000","last_peered":"2024-03-18T14:42:58.324123+0000","last_clean":"2024-03-18T14:42:58.324123+0000","last_became_active":"2024-03-18T14:42:57.326696+0000","last_became_peered":"2024-03-18T14:42:57.326696+0000","last_unstale":"2024-03-18T14:42:58.324123+0000","last_undegraded":"2024-03-18T14:42:58.324123+0000","last_fullsized":"2024-03-18T14:42:58.324123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:03.049979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.307611+0000","last_change":"2024-03-18T14:42:56.323171+0000","last_active":"2024-03-18T14:42:57.307611+0000","last_peered":"2024-03-18T14:42:57.307611+0000","last_clean":"2024-03-18T14:42:57.307611+0000","last_became_active":"2024-03-18T14:42:56.322898+0000","last_became_peered":"2024-03-18T14:42:56.322898+0000","last_unstale":"2024-03-18T14:42:57.307611+0000","last_undegraded":"2024-03-18T14:42:57.307611+0000","last_fullsized":"2024-03-18T14:42:57.307611+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:31.020377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319645+0000","last_change":"2024-03-18T14:42:57.327604+0000","last_active":"2024-03-18T14:42:58.319645+0000","last_peered":"2024-03-18T14:42:58.319645+0000","last_clean":"2024-03-18T14:42:58.319645+0000","last_became_active":"2024-03-18T14:42:57.327196+0000","last_became_peered":"2024-03-18T14:42:57.327196+0000","last_unstale":"2024-03-18T14:42:58.319645+0000","last_undegraded":"2024-03-18T14:42:58.319645+0000","last_fullsized":"2024-03-18T14:42:58.319645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:36.168961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.305414+0000","last_change":"2024-03-18T14:42:56.319304+0000","last_active":"2024-03-18T14:42:57.305414+0000","last_peered":"2024-03-18T14:42:57.305414+0000","last_clean":"2024-03-18T14:42:57.305414+0000","last_became_active":"2024-03-18T14:42:56.318996+0000","last_became_peered":"2024-03-18T14:42:56.318996+0000","last_unstale":"2024-03-18T14:42:57.305414+0000","last_undegraded":"2024-03-18T14:42:57.305414+0000","last_fullsized":"2024-03-18T14:42:57.305414+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:50.599974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330499+0000","last_change":"2024-03-18T14:42:57.327098+0000","last_active":"2024-03-18T14:42:57.330499+0000","last_peered":"2024-03-18T14:42:57.330499+0000","last_clean":"2024-03-18T14:42:57.330499+0000","last_became_active":"2024-03-18T14:42:57.326829+0000","last_became_peered":"2024-03-18T14:42:57.326829+0000","last_unstale":"2024-03-18T14:42:57.330499+0000","last_undegraded":"2024-03-18T14:42:57.330499+0000","last_fullsized":"2024-03-18T14:42:57.330499+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324139+0000","last_change":"2024-03-18T14:42:56.315773+0000","last_active":"2024-03-18T14:42:58.324139+0000","last_peered":"2024-03-18T14:42:58.324139+0000","last_clean":"2024-03-18T14:42:58.324139+0000","last_became_active":"2024-03-18T14:42:56.315441+0000","last_became_peered":"2024-03-18T14:42:56.315441+0000","last_unstale":"2024-03-18T14:42:58.324139+0000","last_undegraded":"2024-03-18T14:42:58.324139+0000","last_fullsized":"2024-03-18T14:42:58.324139+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:36.125139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.329061+0000","last_change":"2024-03-18T14:42:57.317878+0000","last_active":"2024-03-18T14:42:57.329061+0000","last_peered":"2024-03-18T14:42:57.329061+0000","last_clean":"2024-03-18T14:42:57.329061+0000","last_became_active":"2024-03-18T14:42:57.317587+0000","last_became_peered":"2024-03-18T14:42:57.317587+0000","last_unstale":"2024-03-18T14:42:57.329061+0000","last_undegraded":"2024-03-18T14:42:57.329061+0000","last_fullsized":"2024-03-18T14:42:57.329061+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:19.633689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.305522+0000","last_change":"2024-03-18T14:42:56.309300+0000","last_active":"2024-03-18T14:42:57.305522+0000","last_peered":"2024-03-18T14:42:57.305522+0000","last_clean":"2024-03-18T14:42:57.305522+0000","last_became_active":"2024-03-18T14:42:56.309031+0000","last_became_peered":"2024-03-18T14:42:56.309031+0000","last_unstale":"2024-03-18T14:42:57.305522+0000","last_undegraded":"2024-03-18T14:42:57.305522+0000","last_fullsized":"2024-03-18T14:42:57.305522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:51:26.593309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319680+0000","last_change":"2024-03-18T14:42:57.323859+0000","last_active":"2024-03-18T14:42:58.319680+0000","last_peered":"2024-03-18T14:42:58.319680+0000","last_clean":"2024-03-18T14:42:58.319680+0000","last_became_active":"2024-03-18T14:42:57.323530+0000","last_became_peered":"2024-03-18T14:42:57.323530+0000","last_unstale":"2024-03-18T14:42:58.319680+0000","last_undegraded":"2024-03-18T14:42:58.319680+0000","last_fullsized":"2024-03-18T14:42:58.319680+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:07.997004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.319694+0000","last_change":"2024-03-18T14:42:56.322152+0000","last_active":"2024-03-18T14:42:57.319694+0000","last_peered":"2024-03-18T14:42:57.319694+0000","last_clean":"2024-03-18T14:42:57.319694+0000","last_became_active":"2024-03-18T14:42:56.321863+0000","last_became_peered":"2024-03-18T14:42:56.321863+0000","last_unstale":"2024-03-18T14:42:57.319694+0000","last_undegraded":"2024-03-18T14:42:57.319694+0000","last_fullsized":"2024-03-18T14:42:57.319694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:51.796731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334878+0000","last_change":"2024-03-18T14:42:57.328288+0000","last_active":"2024-03-18T14:42:57.334878+0000","last_peered":"2024-03-18T14:42:57.334878+0000","last_clean":"2024-03-18T14:42:57.334878+0000","last_became_active":"2024-03-18T14:42:57.328016+0000","last_became_peered":"2024-03-18T14:42:57.328016+0000","last_unstale":"2024-03-18T14:42:57.334878+0000","last_undegraded":"2024-03-18T14:42:57.334878+0000","last_fullsized":"2024-03-18T14:42:57.334878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.318645+0000","last_change":"2024-03-18T14:42:56.321847+0000","last_active":"2024-03-18T14:42:57.318645+0000","last_peered":"2024-03-18T14:42:57.318645+0000","last_clean":"2024-03-18T14:42:57.318645+0000","last_became_active":"2024-03-18T14:42:56.321532+0000","last_became_peered":"2024-03-18T14:42:56.321532+0000","last_unstale":"2024-03-18T14:42:57.318645+0000","last_undegraded":"2024-03-18T14:42:57.318645+0000","last_fullsized":"2024-03-18T14:42:57.318645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:12.291125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332633+0000","last_change":"2024-03-18T14:42:57.331206+0000","last_active":"2024-03-18T14:42:57.332633+0000","last_peered":"2024-03-18T14:42:57.332633+0000","last_clean":"2024-03-18T14:42:57.332633+0000","last_became_active":"2024-03-18T14:42:57.330863+0000","last_became_peered":"2024-03-18T14:42:57.330863+0000","last_unstale":"2024-03-18T14:42:57.332633+0000","last_undegraded":"2024-03-18T14:42:57.332633+0000","last_fullsized":"2024-03-18T14:42:57.332633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323797+0000","last_change":"2024-03-18T14:42:56.319526+0000","last_active":"2024-03-18T14:42:58.323797+0000","last_peered":"2024-03-18T14:42:58.323797+0000","last_clean":"2024-03-18T14:42:58.323797+0000","last_became_active":"2024-03-18T14:42:56.318876+0000","last_became_peered":"2024-03-18T14:42:56.318876+0000","last_unstale":"2024-03-18T14:42:58.323797+0000","last_undegraded":"2024-03-18T14:42:58.323797+0000","last_fullsized":"2024-03-18T14:42:58.323797+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:00:54.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332957+0000","last_change":"2024-03-18T14:42:57.320372+0000","last_active":"2024-03-18T14:42:57.332957+0000","last_peered":"2024-03-18T14:42:57.332957+0000","last_clean":"2024-03-18T14:42:57.332957+0000","last_became_active":"2024-03-18T14:42:57.320097+0000","last_became_peered":"2024-03-18T14:42:57.320097+0000","last_unstale":"2024-03-18T14:42:57.332957+0000","last_undegraded":"2024-03-18T14:42:57.332957+0000","last_fullsized":"2024-03-18T14:42:57.332957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.313943+0000","last_change":"2024-03-18T14:42:56.323658+0000","last_active":"2024-03-18T14:42:57.313943+0000","last_peered":"2024-03-18T14:42:57.313943+0000","last_clean":"2024-03-18T14:42:57.313943+0000","last_became_active":"2024-03-18T14:42:56.323411+0000","last_became_peered":"2024-03-18T14:42:56.323411+0000","last_unstale":"2024-03-18T14:42:57.313943+0000","last_undegraded":"2024-03-18T14:42:57.313943+0000","last_fullsized":"2024-03-18T14:42:57.313943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:12.140291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.328884+0000","last_change":"2024-03-18T14:42:57.319067+0000","last_active":"2024-03-18T14:42:57.328884+0000","last_peered":"2024-03-18T14:42:57.328884+0000","last_clean":"2024-03-18T14:42:57.328884+0000","last_became_active":"2024-03-18T14:42:57.318817+0000","last_became_peered":"2024-03-18T14:42:57.318817+0000","last_unstale":"2024-03-18T14:42:57.328884+0000","last_undegraded":"2024-03-18T14:42:57.328884+0000","last_fullsized":"2024-03-18T14:42:57.328884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:11.940592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319402+0000","last_change":"2024-03-18T14:42:56.315423+0000","last_active":"2024-03-18T14:42:58.319402+0000","last_peered":"2024-03-18T14:42:58.319402+0000","last_clean":"2024-03-18T14:42:58.319402+0000","last_became_active":"2024-03-18T14:42:56.314846+0000","last_became_peered":"2024-03-18T14:42:56.314846+0000","last_unstale":"2024-03-18T14:42:58.319402+0000","last_undegraded":"2024-03-18T14:42:58.319402+0000","last_fullsized":"2024-03-18T14:42:58.319402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:28.354432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324041+0000","last_change":"2024-03-18T14:42:57.322324+0000","last_active":"2024-03-18T14:42:58.324041+0000","last_peered":"2024-03-18T14:42:58.324041+0000","last_clean":"2024-03-18T14:42:58.324041+0000","last_became_active":"2024-03-18T14:42:57.322044+0000","last_became_peered":"2024-03-18T14:42:57.322044+0000","last_unstale":"2024-03-18T14:42:58.324041+0000","last_undegraded":"2024-03-18T14:42:58.324041+0000","last_fullsized":"2024-03-18T14:42:58.324041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:52:36.862870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.308250+0000","last_change":"2024-03-18T14:42:56.324502+0000","last_active":"2024-03-18T14:42:57.308250+0000","last_peered":"2024-03-18T14:42:57.308250+0000","last_clean":"2024-03-18T14:42:57.308250+0000","last_became_active":"2024-03-18T14:42:56.324303+0000","last_became_peered":"2024-03-18T14:42:56.324303+0000","last_unstale":"2024-03-18T14:42:57.308250+0000","last_undegraded":"2024-03-18T14:42:57.308250+0000","last_fullsized":"2024-03-18T14:42:57.308250+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:21.948479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.306422+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:42:57.306422+0000","last_peered":"2024-03-18T14:42:57.306422+0000","last_clean":"2024-03-18T14:42:57.306422+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:42:57.306422+0000","last_undegraded":"2024-03-18T14:42:57.306422+0000","last_fullsized":"2024-03-18T14:42:57.306422+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":780065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":189661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1036288,"data_stored":778417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:43:00.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:43:00.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.458+0000 7f111651b640 1 Processor -- start 2024-03-18T14:43:00.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.458+0000 7f111651b640 1 -- start start 2024-03-18T14:43:00.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.459+0000 7f111651b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f111651b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11100d1d90 0x7f11100cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f111651b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11100cfcd0 0x7f11100ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f111651b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f11100cc870 con 0x7f11100d1d90 2024-03-18T14:43:00.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f111651b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f11100d0500 con 0x7f11100cfcd0 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f111651b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f11100d07e0 con 0x7f111015bd40 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11100d1d90 0x7f11100cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11100d1d90 0x7f11100cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58006/0 (socket says 172.21.15.6:58006) 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 -- 172.21.15.6:0/1913618877 learned_addr learned my addr 172.21.15.6:0/1913618877 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110ffff640 1 --2- 172.21.15.6:0/1913618877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f1114ab8640 1 --2- 172.21.15.6:0/1913618877 >> v2:172.21.15.92:3300/0 conn(0x7f11100cfcd0 0x7f11100ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 -- 172.21.15.6:0/1913618877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 msgr2=0x7f11100d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 --2- 172.21.15.6:0/1913618877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 -- 172.21.15.6:0/1913618877 >> v2:172.21.15.92:3300/0 conn(0x7f11100cfcd0 msgr2=0x7f11100ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 --2- 172.21.15.6:0/1913618877 >> v2:172.21.15.92:3300/0 conn(0x7f11100cfcd0 0x7f11100ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.460+0000 7f110f7fe640 1 -- 172.21.15.6:0/1913618877 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11100ccb10 con 0x7f11100d1d90 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f110f7fe640 1 --2- 172.21.15.6:0/1913618877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100d1d90 0x7f11100cf790 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f11100cf5e0 tx=0x7f10f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e521a780748138f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f110effd640 1 -- 172.21.15.6:0/1913618877 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8035020 con 0x7f11100d1d90 2024-03-18T14:43:00.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f110effd640 1 -- 172.21.15.6:0/1913618877 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f8031870 con 0x7f11100d1d90 2024-03-18T14:43:00.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f110effd640 1 -- 172.21.15.6:0/1913618877 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8031c40 con 0x7f11100d1d90 2024-03-18T14:43:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f111651b640 1 -- 172.21.15.6:0/1913618877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100d1d90 msgr2=0x7f11100cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f111651b640 1 --2- 172.21.15.6:0/1913618877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100d1d90 0x7f11100cf790 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f11100cf5e0 tx=0x7f10f802b910 comp rx=0 tx=0).stop 2024-03-18T14:43:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f111651b640 1 -- 172.21.15.6:0/1913618877 shutdown_connections 2024-03-18T14:43:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f111651b640 1 --2- 172.21.15.6:0/1913618877 >> v2:172.21.15.92:3300/0 conn(0x7f11100cfcd0 0x7f11100ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f111651b640 1 --2- 172.21.15.6:0/1913618877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100d1d90 0x7f11100cf790 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f111651b640 1 --2- 172.21.15.6:0/1913618877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.461+0000 7f111651b640 1 -- 172.21.15.6:0/1913618877 >> 172.21.15.6:0/1913618877 conn(0x7f111009ae20 msgr2=0x7f111008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.462+0000 7f111651b640 1 -- 172.21.15.6:0/1913618877 shutdown_connections 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.462+0000 7f111651b640 1 -- 172.21.15.6:0/1913618877 wait complete. 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.463+0000 7f111651b640 1 Processor -- start 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.463+0000 7f111651b640 1 -- start start 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100cfcd0 0x7f1110158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11100d1d90 0x7f111014ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f11100d0c40 con 0x7f11100cfcd0 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f11100d0ac0 con 0x7f11100d1d90 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f11100d0160 con 0x7f111015bd40 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100cfcd0 0x7f1110158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100cfcd0 0x7f1110158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58016/0 (socket says 172.21.15.6:58016) 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 -- 172.21.15.6:0/1294657807 learned_addr learned my addr 172.21.15.6:0/1294657807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f1114ab8640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110f7fe640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11100d1d90 0x7f111014ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 -- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 msgr2=0x7f11100b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 -- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11100d1d90 msgr2=0x7f111014ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11100d1d90 0x7f111014ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 -- 172.21.15.6:0/1294657807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111015f220 con 0x7f11100cfcd0 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f1114ab8640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110f7fe640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11100d1d90 0x7f111014ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110ffff640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100cfcd0 0x7f1110158310 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f110000d5f0 tx=0x7f110000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110cff9640 1 -- 172.21.15.6:0/1294657807 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11000125f0 con 0x7f11100cfcd0 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110cff9640 1 -- 172.21.15.6:0/1294657807 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1100019070 con 0x7f11100cfcd0 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111015ccf0 con 0x7f11100cfcd0 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f110cff9640 1 -- 172.21.15.6:0/1294657807 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11000129d0 con 0x7f11100cfcd0 2024-03-18T14:43:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.464+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111014f2f0 con 0x7f11100cfcd0 2024-03-18T14:43:00.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.465+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111009fa90 con 0x7f11100cfcd0 2024-03-18T14:43:00.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.466+0000 7f110cff9640 1 -- 172.21.15.6:0/1294657807 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1100012b70 con 0x7f11100cfcd0 2024-03-18T14:43:00.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.467+0000 7f110cff9640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10f006bbd0 0x7f10f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.467+0000 7f110cff9640 1 -- 172.21.15.6:0/1294657807 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f110008af60 con 0x7f11100cfcd0 2024-03-18T14:43:00.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.467+0000 7f110f7fe640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10f006bbd0 0x7f10f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.467+0000 7f110f7fe640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10f006bbd0 0x7f10f006e090 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f10f8036000 tx=0x7f10f803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.471+0000 7f110cff9640 1 -- 172.21.15.6:0/1294657807 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1100059950 con 0x7f11100cfcd0 2024-03-18T14:43:00.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.649+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f11100b5b50 con 0x7f10f006bbd0 2024-03-18T14:43:00.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.655+0000 7f110cff9640 1 -- 172.21.15.6:0/1294657807 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251914 (secure 0 0 0) 0x7f11100b5b50 con 0x7f10f006bbd0 2024-03-18T14:43:00.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10f006bbd0 msgr2=0x7f10f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10f006bbd0 0x7f10f006e090 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f10f8036000 tx=0x7f10f803c040 comp rx=0 tx=0).stop 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100cfcd0 msgr2=0x7f1110158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100cfcd0 0x7f1110158310 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f110000d5f0 tx=0x7f110000dab0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 shutdown_connections 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10f006bbd0 0x7f10f006e090 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f111015bd40 0x7f11100b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11100d1d90 0x7f111014ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 --2- 172.21.15.6:0/1294657807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100cfcd0 0x7f1110158310 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 >> 172.21.15.6:0/1294657807 conn(0x7f111009ae20 msgr2=0x7f11100b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:00.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 shutdown_connections 2024-03-18T14:43:00.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.658+0000 7f111651b640 1 -- 172.21.15.6:0/1294657807 wait complete. 2024-03-18T14:43:00.670 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-03-18T14:42:59.396878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220824,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753896,"statfs":{"total":773094113280,"available":772867989504,"internally_reserved":0,"allocated":4759552,"data_stored":2689676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.100814"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319901+0000","last_change":"2024-03-18T14:42:57.331932+0000","last_active":"2024-03-18T14:42:58.319901+0000","last_peered":"2024-03-18T14:42:58.319901+0000","last_clean":"2024-03-18T14:42:58.319901+0000","last_became_active":"2024-03-18T14:42:57.331648+0000","last_became_peered":"2024-03-18T14:42:57.331648+0000","last_unstale":"2024-03-18T14:42:58.319901+0000","last_undegraded":"2024-03-18T14:42:58.319901+0000","last_fullsized":"2024-03-18T14:42:58.319901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:43.723920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319670+0000","last_change":"2024-03-18T14:42:57.314223+0000","last_active":"2024-03-18T14:42:58.319670+0000","last_peered":"2024-03-18T14:42:58.319670+0000","last_clean":"2024-03-18T14:42:58.319670+0000","last_became_active":"2024-03-18T14:42:57.313928+0000","last_became_peered":"2024-03-18T14:42:57.313928+0000","last_unstale":"2024-03-18T14:42:58.319670+0000","last_undegraded":"2024-03-18T14:42:58.319670+0000","last_fullsized":"2024-03-18T14:42:58.319670+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:34.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334939+0000","last_change":"2024-03-18T14:42:57.326257+0000","last_active":"2024-03-18T14:42:57.334939+0000","last_peered":"2024-03-18T14:42:57.334939+0000","last_clean":"2024-03-18T14:42:57.334939+0000","last_became_active":"2024-03-18T14:42:57.325962+0000","last_became_peered":"2024-03-18T14:42:57.325962+0000","last_unstale":"2024-03-18T14:42:57.334939+0000","last_undegraded":"2024-03-18T14:42:57.334939+0000","last_fullsized":"2024-03-18T14:42:57.334939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330597+0000","last_change":"2024-03-18T14:42:57.323720+0000","last_active":"2024-03-18T14:42:57.330597+0000","last_peered":"2024-03-18T14:42:57.330597+0000","last_clean":"2024-03-18T14:42:57.330597+0000","last_became_active":"2024-03-18T14:42:57.322969+0000","last_became_peered":"2024-03-18T14:42:57.322969+0000","last_unstale":"2024-03-18T14:42:57.330597+0000","last_undegraded":"2024-03-18T14:42:57.330597+0000","last_fullsized":"2024-03-18T14:42:57.330597+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:45.592453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.324807+0000","last_change":"2024-03-18T14:42:57.321988+0000","last_active":"2024-03-18T14:42:57.324807+0000","last_peered":"2024-03-18T14:42:57.324807+0000","last_clean":"2024-03-18T14:42:57.324807+0000","last_became_active":"2024-03-18T14:42:57.321648+0000","last_became_peered":"2024-03-18T14:42:57.321648+0000","last_unstale":"2024-03-18T14:42:57.324807+0000","last_undegraded":"2024-03-18T14:42:57.324807+0000","last_fullsized":"2024-03-18T14:42:57.324807+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:19:34.015756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334767+0000","last_change":"2024-03-18T14:42:57.331614+0000","last_active":"2024-03-18T14:42:57.334767+0000","last_peered":"2024-03-18T14:42:57.334767+0000","last_clean":"2024-03-18T14:42:57.334767+0000","last_became_active":"2024-03-18T14:42:57.331321+0000","last_became_peered":"2024-03-18T14:42:57.331321+0000","last_unstale":"2024-03-18T14:42:57.334767+0000","last_undegraded":"2024-03-18T14:42:57.334767+0000","last_fullsized":"2024-03-18T14:42:57.334767+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324119+0000","last_change":"2024-03-18T14:42:57.324942+0000","last_active":"2024-03-18T14:42:58.324119+0000","last_peered":"2024-03-18T14:42:58.324119+0000","last_clean":"2024-03-18T14:42:58.324119+0000","last_became_active":"2024-03-18T14:42:57.323179+0000","last_became_peered":"2024-03-18T14:42:57.323179+0000","last_unstale":"2024-03-18T14:42:58.324119+0000","last_undegraded":"2024-03-18T14:42:58.324119+0000","last_fullsized":"2024-03-18T14:42:58.324119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:51:26.159733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319426+0000","last_change":"2024-03-18T14:42:57.334380+0000","last_active":"2024-03-18T14:42:58.319426+0000","last_peered":"2024-03-18T14:42:58.319426+0000","last_clean":"2024-03-18T14:42:58.319426+0000","last_became_active":"2024-03-18T14:42:57.334044+0000","last_became_peered":"2024-03-18T14:42:57.334044+0000","last_unstale":"2024-03-18T14:42:58.319426+0000","last_undegraded":"2024-03-18T14:42:58.319426+0000","last_fullsized":"2024-03-18T14:42:58.319426+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:03.049979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.335008+0000","last_change":"2024-03-18T14:42:57.325974+0000","last_active":"2024-03-18T14:42:57.335008+0000","last_peered":"2024-03-18T14:42:57.335008+0000","last_clean":"2024-03-18T14:42:57.335008+0000","last_became_active":"2024-03-18T14:42:57.325631+0000","last_became_peered":"2024-03-18T14:42:57.325631+0000","last_unstale":"2024-03-18T14:42:57.335008+0000","last_undegraded":"2024-03-18T14:42:57.335008+0000","last_fullsized":"2024-03-18T14:42:57.335008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324075+0000","last_change":"2024-03-18T14:42:57.329271+0000","last_active":"2024-03-18T14:42:58.324075+0000","last_peered":"2024-03-18T14:42:58.324075+0000","last_clean":"2024-03-18T14:42:58.324075+0000","last_became_active":"2024-03-18T14:42:57.329010+0000","last_became_peered":"2024-03-18T14:42:57.329010+0000","last_unstale":"2024-03-18T14:42:58.324075+0000","last_undegraded":"2024-03-18T14:42:58.324075+0000","last_fullsized":"2024-03-18T14:42:58.324075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:58.859788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324269+0000","last_change":"2024-03-18T14:42:57.322920+0000","last_active":"2024-03-18T14:42:58.324269+0000","last_peered":"2024-03-18T14:42:58.324269+0000","last_clean":"2024-03-18T14:42:58.324269+0000","last_became_active":"2024-03-18T14:42:57.322640+0000","last_became_peered":"2024-03-18T14:42:57.322640+0000","last_unstale":"2024-03-18T14:42:58.324269+0000","last_undegraded":"2024-03-18T14:42:58.324269+0000","last_fullsized":"2024-03-18T14:42:58.324269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:36.168961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334989+0000","last_change":"2024-03-18T14:42:57.328727+0000","last_active":"2024-03-18T14:42:57.334989+0000","last_peered":"2024-03-18T14:42:57.334989+0000","last_clean":"2024-03-18T14:42:57.334989+0000","last_became_active":"2024-03-18T14:42:57.328372+0000","last_became_peered":"2024-03-18T14:42:57.328372+0000","last_unstale":"2024-03-18T14:42:57.334989+0000","last_undegraded":"2024-03-18T14:42:57.334989+0000","last_fullsized":"2024-03-18T14:42:57.334989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324057+0000","last_change":"2024-03-18T14:42:57.331522+0000","last_active":"2024-03-18T14:42:58.324057+0000","last_peered":"2024-03-18T14:42:58.324057+0000","last_clean":"2024-03-18T14:42:58.324057+0000","last_became_active":"2024-03-18T14:42:57.329664+0000","last_became_peered":"2024-03-18T14:42:57.329664+0000","last_unstale":"2024-03-18T14:42:58.324057+0000","last_undegraded":"2024-03-18T14:42:58.324057+0000","last_fullsized":"2024-03-18T14:42:58.324057+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:51:04.081506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333450+0000","last_change":"2024-03-18T14:42:57.329322+0000","last_active":"2024-03-18T14:42:57.333450+0000","last_peered":"2024-03-18T14:42:57.333450+0000","last_clean":"2024-03-18T14:42:57.333450+0000","last_became_active":"2024-03-18T14:42:57.329033+0000","last_became_peered":"2024-03-18T14:42:57.329033+0000","last_unstale":"2024-03-18T14:42:57.333450+0000","last_undegraded":"2024-03-18T14:42:57.333450+0000","last_fullsized":"2024-03-18T14:42:57.333450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:36.914241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324154+0000","last_change":"2024-03-18T14:42:57.324633+0000","last_active":"2024-03-18T14:42:58.324154+0000","last_peered":"2024-03-18T14:42:58.324154+0000","last_clean":"2024-03-18T14:42:58.324154+0000","last_became_active":"2024-03-18T14:42:57.322820+0000","last_became_peered":"2024-03-18T14:42:57.322820+0000","last_unstale":"2024-03-18T14:42:58.324154+0000","last_undegraded":"2024-03-18T14:42:58.324154+0000","last_fullsized":"2024-03-18T14:42:58.324154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:33:17.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.326682+0000","last_change":"2024-03-18T14:42:57.320474+0000","last_active":"2024-03-18T14:42:57.326682+0000","last_peered":"2024-03-18T14:42:57.326682+0000","last_clean":"2024-03-18T14:42:57.326682+0000","last_became_active":"2024-03-18T14:42:57.320076+0000","last_became_peered":"2024-03-18T14:42:57.320076+0000","last_unstale":"2024-03-18T14:42:57.326682+0000","last_undegraded":"2024-03-18T14:42:57.326682+0000","last_fullsized":"2024-03-18T14:42:57.326682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:19.966646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.327211+0000","last_change":"2024-03-18T14:42:57.323775+0000","last_active":"2024-03-18T14:42:57.327211+0000","last_peered":"2024-03-18T14:42:57.327211+0000","last_clean":"2024-03-18T14:42:57.327211+0000","last_became_active":"2024-03-18T14:42:57.323496+0000","last_became_peered":"2024-03-18T14:42:57.323496+0000","last_unstale":"2024-03-18T14:42:57.327211+0000","last_undegraded":"2024-03-18T14:42:57.327211+0000","last_fullsized":"2024-03-18T14:42:57.327211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:29.356561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323602+0000","last_change":"2024-03-18T14:42:57.328474+0000","last_active":"2024-03-18T14:42:58.323602+0000","last_peered":"2024-03-18T14:42:58.323602+0000","last_clean":"2024-03-18T14:42:58.323602+0000","last_became_active":"2024-03-18T14:42:57.328224+0000","last_became_peered":"2024-03-18T14:42:57.328224+0000","last_unstale":"2024-03-18T14:42:58.323602+0000","last_undegraded":"2024-03-18T14:42:58.323602+0000","last_fullsized":"2024-03-18T14:42:58.323602+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330433+0000","last_change":"2024-03-18T14:42:57.323110+0000","last_active":"2024-03-18T14:42:57.330433+0000","last_peered":"2024-03-18T14:42:57.330433+0000","last_clean":"2024-03-18T14:42:57.330433+0000","last_became_active":"2024-03-18T14:42:57.322800+0000","last_became_peered":"2024-03-18T14:42:57.322800+0000","last_unstale":"2024-03-18T14:42:57.330433+0000","last_undegraded":"2024-03-18T14:42:57.330433+0000","last_fullsized":"2024-03-18T14:42:57.330433+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:21:21.131435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323194+0000","last_change":"2024-03-18T14:42:57.318916+0000","last_active":"2024-03-18T14:42:58.323194+0000","last_peered":"2024-03-18T14:42:58.323194+0000","last_clean":"2024-03-18T14:42:58.323194+0000","last_became_active":"2024-03-18T14:42:57.318351+0000","last_became_peered":"2024-03-18T14:42:57.318351+0000","last_unstale":"2024-03-18T14:42:58.323194+0000","last_undegraded":"2024-03-18T14:42:58.323194+0000","last_fullsized":"2024-03-18T14:42:58.323194+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:13.142420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324145+0000","last_change":"2024-03-18T14:42:57.322170+0000","last_active":"2024-03-18T14:42:58.324145+0000","last_peered":"2024-03-18T14:42:58.324145+0000","last_clean":"2024-03-18T14:42:58.324145+0000","last_became_active":"2024-03-18T14:42:57.321891+0000","last_became_peered":"2024-03-18T14:42:57.321891+0000","last_unstale":"2024-03-18T14:42:58.324145+0000","last_undegraded":"2024-03-18T14:42:58.324145+0000","last_fullsized":"2024-03-18T14:42:58.324145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:22.016912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333247+0000","last_change":"2024-03-18T14:42:57.325931+0000","last_active":"2024-03-18T14:42:57.333247+0000","last_peered":"2024-03-18T14:42:57.333247+0000","last_clean":"2024-03-18T14:42:57.333247+0000","last_became_active":"2024-03-18T14:42:57.325639+0000","last_became_peered":"2024-03-18T14:42:57.325639+0000","last_unstale":"2024-03-18T14:42:57.333247+0000","last_undegraded":"2024-03-18T14:42:57.333247+0000","last_fullsized":"2024-03-18T14:42:57.333247+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.325572+0000","last_change":"2024-03-18T14:42:57.316559+0000","last_active":"2024-03-18T14:42:57.325572+0000","last_peered":"2024-03-18T14:42:57.325572+0000","last_clean":"2024-03-18T14:42:57.325572+0000","last_became_active":"2024-03-18T14:42:57.315949+0000","last_became_peered":"2024-03-18T14:42:57.315949+0000","last_unstale":"2024-03-18T14:42:57.325572+0000","last_undegraded":"2024-03-18T14:42:57.325572+0000","last_fullsized":"2024-03-18T14:42:57.325572+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.314576+0000","last_change":"2024-03-18T14:42:57.312017+0000","last_active":"2024-03-18T14:42:57.314576+0000","last_peered":"2024-03-18T14:42:57.314576+0000","last_clean":"2024-03-18T14:42:57.314576+0000","last_became_active":"2024-03-18T14:42:57.311706+0000","last_became_peered":"2024-03-18T14:42:57.311706+0000","last_unstale":"2024-03-18T14:42:57.314576+0000","last_undegraded":"2024-03-18T14:42:57.314576+0000","last_fullsized":"2024-03-18T14:42:57.314576+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:05.687491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323967+0000","last_change":"2024-03-18T14:42:57.324082+0000","last_active":"2024-03-18T14:42:58.323967+0000","last_peered":"2024-03-18T14:42:58.323967+0000","last_clean":"2024-03-18T14:42:58.323967+0000","last_became_active":"2024-03-18T14:42:57.323801+0000","last_became_peered":"2024-03-18T14:42:57.323801+0000","last_unstale":"2024-03-18T14:42:58.323967+0000","last_undegraded":"2024-03-18T14:42:58.323967+0000","last_fullsized":"2024-03-18T14:42:58.323967+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330467+0000","last_change":"2024-03-18T14:42:57.327616+0000","last_active":"2024-03-18T14:42:57.330467+0000","last_peered":"2024-03-18T14:42:57.330467+0000","last_clean":"2024-03-18T14:42:57.330467+0000","last_became_active":"2024-03-18T14:42:57.327327+0000","last_became_peered":"2024-03-18T14:42:57.327327+0000","last_unstale":"2024-03-18T14:42:57.330467+0000","last_undegraded":"2024-03-18T14:42:57.330467+0000","last_fullsized":"2024-03-18T14:42:57.330467+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323346+0000","last_change":"2024-03-18T14:42:57.313584+0000","last_active":"2024-03-18T14:42:58.323346+0000","last_peered":"2024-03-18T14:42:58.323346+0000","last_clean":"2024-03-18T14:42:58.323346+0000","last_became_active":"2024-03-18T14:42:57.313306+0000","last_became_peered":"2024-03-18T14:42:57.313306+0000","last_unstale":"2024-03-18T14:42:58.323346+0000","last_undegraded":"2024-03-18T14:42:58.323346+0000","last_fullsized":"2024-03-18T14:42:58.323346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:19.877972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324190+0000","last_change":"2024-03-18T14:42:57.321881+0000","last_active":"2024-03-18T14:42:58.324190+0000","last_peered":"2024-03-18T14:42:58.324190+0000","last_clean":"2024-03-18T14:42:58.324190+0000","last_became_active":"2024-03-18T14:42:57.321599+0000","last_became_peered":"2024-03-18T14:42:57.321599+0000","last_unstale":"2024-03-18T14:42:58.324190+0000","last_undegraded":"2024-03-18T14:42:58.324190+0000","last_fullsized":"2024-03-18T14:42:58.324190+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:11.493713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323437+0000","last_change":"2024-03-18T14:42:57.318718+0000","last_active":"2024-03-18T14:42:58.323437+0000","last_peered":"2024-03-18T14:42:58.323437+0000","last_clean":"2024-03-18T14:42:58.323437+0000","last_became_active":"2024-03-18T14:42:57.318427+0000","last_became_peered":"2024-03-18T14:42:57.318427+0000","last_unstale":"2024-03-18T14:42:58.323437+0000","last_undegraded":"2024-03-18T14:42:58.323437+0000","last_fullsized":"2024-03-18T14:42:58.323437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323725+0000","last_change":"2024-03-18T14:42:57.314465+0000","last_active":"2024-03-18T14:42:58.323725+0000","last_peered":"2024-03-18T14:42:58.323725+0000","last_clean":"2024-03-18T14:42:58.323725+0000","last_became_active":"2024-03-18T14:42:57.314223+0000","last_became_peered":"2024-03-18T14:42:57.314223+0000","last_unstale":"2024-03-18T14:42:58.323725+0000","last_undegraded":"2024-03-18T14:42:58.323725+0000","last_fullsized":"2024-03-18T14:42:58.323725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.325553+0000","last_change":"2024-03-18T14:42:57.316110+0000","last_active":"2024-03-18T14:42:57.325553+0000","last_peered":"2024-03-18T14:42:57.325553+0000","last_clean":"2024-03-18T14:42:57.325553+0000","last_became_active":"2024-03-18T14:42:57.315831+0000","last_became_peered":"2024-03-18T14:42:57.315831+0000","last_unstale":"2024-03-18T14:42:57.325553+0000","last_undegraded":"2024-03-18T14:42:57.325553+0000","last_fullsized":"2024-03-18T14:42:57.325553+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334640+0000","last_change":"2024-03-18T14:42:57.331604+0000","last_active":"2024-03-18T14:42:57.334640+0000","last_peered":"2024-03-18T14:42:57.334640+0000","last_clean":"2024-03-18T14:42:57.334640+0000","last_became_active":"2024-03-18T14:42:57.331283+0000","last_became_peered":"2024-03-18T14:42:57.331283+0000","last_unstale":"2024-03-18T14:42:57.334640+0000","last_undegraded":"2024-03-18T14:42:57.334640+0000","last_fullsized":"2024-03-18T14:42:57.334640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.327119+0000","last_change":"2024-03-18T14:42:57.325069+0000","last_active":"2024-03-18T14:42:57.327119+0000","last_peered":"2024-03-18T14:42:57.327119+0000","last_clean":"2024-03-18T14:42:57.327119+0000","last_became_active":"2024-03-18T14:42:57.324466+0000","last_became_peered":"2024-03-18T14:42:57.324466+0000","last_unstale":"2024-03-18T14:42:57.327119+0000","last_undegraded":"2024-03-18T14:42:57.327119+0000","last_fullsized":"2024-03-18T14:42:57.327119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:07.247126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323957+0000","last_change":"2024-03-18T14:42:56.316599+0000","last_active":"2024-03-18T14:42:58.323957+0000","last_peered":"2024-03-18T14:42:58.323957+0000","last_clean":"2024-03-18T14:42:58.323957+0000","last_became_active":"2024-03-18T14:42:56.316212+0000","last_became_peered":"2024-03-18T14:42:56.316212+0000","last_unstale":"2024-03-18T14:42:58.323957+0000","last_undegraded":"2024-03-18T14:42:58.323957+0000","last_fullsized":"2024-03-18T14:42:58.323957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:04:45.807569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330528+0000","last_change":"2024-03-18T14:42:57.326929+0000","last_active":"2024-03-18T14:42:57.330528+0000","last_peered":"2024-03-18T14:42:57.330528+0000","last_clean":"2024-03-18T14:42:57.330528+0000","last_became_active":"2024-03-18T14:42:57.326648+0000","last_became_peered":"2024-03-18T14:42:57.326648+0000","last_unstale":"2024-03-18T14:42:57.330528+0000","last_undegraded":"2024-03-18T14:42:57.330528+0000","last_fullsized":"2024-03-18T14:42:57.330528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:19.877972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.329215+0000","last_change":"2024-03-18T14:42:56.324129+0000","last_active":"2024-03-18T14:42:57.329215+0000","last_peered":"2024-03-18T14:42:57.329215+0000","last_clean":"2024-03-18T14:42:57.329215+0000","last_became_active":"2024-03-18T14:42:56.323848+0000","last_became_peered":"2024-03-18T14:42:56.323848+0000","last_unstale":"2024-03-18T14:42:57.329215+0000","last_undegraded":"2024-03-18T14:42:57.329215+0000","last_fullsized":"2024-03-18T14:42:57.329215+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:09.445509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333709+0000","last_change":"2024-03-18T14:42:57.331045+0000","last_active":"2024-03-18T14:42:57.333709+0000","last_peered":"2024-03-18T14:42:57.333709+0000","last_clean":"2024-03-18T14:42:57.333709+0000","last_became_active":"2024-03-18T14:42:57.330772+0000","last_became_peered":"2024-03-18T14:42:57.330772+0000","last_unstale":"2024-03-18T14:42:57.333709+0000","last_undegraded":"2024-03-18T14:42:57.333709+0000","last_fullsized":"2024-03-18T14:42:57.333709+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:37.127269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333492+0000","last_change":"2024-03-18T14:42:56.318507+0000","last_active":"2024-03-18T14:42:57.333492+0000","last_peered":"2024-03-18T14:42:57.333492+0000","last_clean":"2024-03-18T14:42:57.333492+0000","last_became_active":"2024-03-18T14:42:56.317461+0000","last_became_peered":"2024-03-18T14:42:56.317461+0000","last_unstale":"2024-03-18T14:42:57.333492+0000","last_undegraded":"2024-03-18T14:42:57.333492+0000","last_fullsized":"2024-03-18T14:42:57.333492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:51.796731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319858+0000","last_change":"2024-03-18T14:42:57.331698+0000","last_active":"2024-03-18T14:42:58.319858+0000","last_peered":"2024-03-18T14:42:58.319858+0000","last_clean":"2024-03-18T14:42:58.319858+0000","last_became_active":"2024-03-18T14:42:57.331247+0000","last_became_peered":"2024-03-18T14:42:57.331247+0000","last_unstale":"2024-03-18T14:42:58.319858+0000","last_undegraded":"2024-03-18T14:42:58.319858+0000","last_fullsized":"2024-03-18T14:42:58.319858+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:58.272509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.336687+0000","last_change":"2024-03-18T14:42:56.319748+0000","last_active":"2024-03-18T14:42:57.336687+0000","last_peered":"2024-03-18T14:42:57.336687+0000","last_clean":"2024-03-18T14:42:57.336687+0000","last_became_active":"2024-03-18T14:42:56.319544+0000","last_became_peered":"2024-03-18T14:42:56.319544+0000","last_unstale":"2024-03-18T14:42:57.336687+0000","last_undegraded":"2024-03-18T14:42:57.336687+0000","last_fullsized":"2024-03-18T14:42:57.336687+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:12.291125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334829+0000","last_change":"2024-03-18T14:42:57.325466+0000","last_active":"2024-03-18T14:42:57.334829+0000","last_peered":"2024-03-18T14:42:57.334829+0000","last_clean":"2024-03-18T14:42:57.334829+0000","last_became_active":"2024-03-18T14:42:57.325123+0000","last_became_peered":"2024-03-18T14:42:57.325123+0000","last_unstale":"2024-03-18T14:42:57.334829+0000","last_undegraded":"2024-03-18T14:42:57.334829+0000","last_fullsized":"2024-03-18T14:42:57.334829+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:13.142420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324339+0000","last_change":"2024-03-18T14:42:56.318873+0000","last_active":"2024-03-18T14:42:58.324339+0000","last_peered":"2024-03-18T14:42:58.324339+0000","last_clean":"2024-03-18T14:42:58.324339+0000","last_became_active":"2024-03-18T14:42:56.318585+0000","last_became_peered":"2024-03-18T14:42:56.318585+0000","last_unstale":"2024-03-18T14:42:58.324339+0000","last_undegraded":"2024-03-18T14:42:58.324339+0000","last_fullsized":"2024-03-18T14:42:58.324339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:25.894064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324320+0000","last_change":"2024-03-18T14:42:57.325189+0000","last_active":"2024-03-18T14:42:58.324320+0000","last_peered":"2024-03-18T14:42:58.324320+0000","last_clean":"2024-03-18T14:42:58.324320+0000","last_became_active":"2024-03-18T14:42:57.324545+0000","last_became_peered":"2024-03-18T14:42:57.324545+0000","last_unstale":"2024-03-18T14:42:58.324320+0000","last_undegraded":"2024-03-18T14:42:58.324320+0000","last_fullsized":"2024-03-18T14:42:58.324320+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:34.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.336457+0000","last_change":"2024-03-18T14:42:56.323154+0000","last_active":"2024-03-18T14:42:57.336457+0000","last_peered":"2024-03-18T14:42:57.336457+0000","last_clean":"2024-03-18T14:42:57.336457+0000","last_became_active":"2024-03-18T14:42:56.322880+0000","last_became_peered":"2024-03-18T14:42:56.322880+0000","last_unstale":"2024-03-18T14:42:57.336457+0000","last_undegraded":"2024-03-18T14:42:57.336457+0000","last_fullsized":"2024-03-18T14:42:57.336457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:18.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332979+0000","last_change":"2024-03-18T14:42:57.322920+0000","last_active":"2024-03-18T14:42:57.332979+0000","last_peered":"2024-03-18T14:42:57.332979+0000","last_clean":"2024-03-18T14:42:57.332979+0000","last_became_active":"2024-03-18T14:42:57.322655+0000","last_became_peered":"2024-03-18T14:42:57.322655+0000","last_unstale":"2024-03-18T14:42:57.332979+0000","last_undegraded":"2024-03-18T14:42:57.332979+0000","last_fullsized":"2024-03-18T14:42:57.332979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.312580+0000","last_change":"2024-03-18T14:42:56.317870+0000","last_active":"2024-03-18T14:42:57.312580+0000","last_peered":"2024-03-18T14:42:57.312580+0000","last_clean":"2024-03-18T14:42:57.312580+0000","last_became_active":"2024-03-18T14:42:56.317610+0000","last_became_peered":"2024-03-18T14:42:56.317610+0000","last_unstale":"2024-03-18T14:42:57.312580+0000","last_undegraded":"2024-03-18T14:42:57.312580+0000","last_fullsized":"2024-03-18T14:42:57.312580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:09.445509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319446+0000","last_change":"2024-03-18T14:42:57.328371+0000","last_active":"2024-03-18T14:42:58.319446+0000","last_peered":"2024-03-18T14:42:58.319446+0000","last_clean":"2024-03-18T14:42:58.319446+0000","last_became_active":"2024-03-18T14:42:57.328065+0000","last_became_peered":"2024-03-18T14:42:57.328065+0000","last_unstale":"2024-03-18T14:42:58.319446+0000","last_undegraded":"2024-03-18T14:42:58.319446+0000","last_fullsized":"2024-03-18T14:42:58.319446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:22.016912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.313104+0000","last_change":"2024-03-18T14:42:56.316817+0000","last_active":"2024-03-18T14:42:57.313104+0000","last_peered":"2024-03-18T14:42:57.313104+0000","last_clean":"2024-03-18T14:42:57.313104+0000","last_became_active":"2024-03-18T14:42:56.316547+0000","last_became_peered":"2024-03-18T14:42:56.316547+0000","last_unstale":"2024-03-18T14:42:57.313104+0000","last_undegraded":"2024-03-18T14:42:57.313104+0000","last_fullsized":"2024-03-18T14:42:57.313104+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:51.796731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330575+0000","last_change":"2024-03-18T14:42:57.323438+0000","last_active":"2024-03-18T14:42:57.330575+0000","last_peered":"2024-03-18T14:42:57.330575+0000","last_clean":"2024-03-18T14:42:57.330575+0000","last_became_active":"2024-03-18T14:42:57.322586+0000","last_became_peered":"2024-03-18T14:42:57.322586+0000","last_unstale":"2024-03-18T14:42:57.330575+0000","last_undegraded":"2024-03-18T14:42:57.330575+0000","last_fullsized":"2024-03-18T14:42:57.330575+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:04:46.809699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319907+0000","last_change":"2024-03-18T14:42:56.318787+0000","last_active":"2024-03-18T14:42:58.319907+0000","last_peered":"2024-03-18T14:42:58.319907+0000","last_clean":"2024-03-18T14:42:58.319907+0000","last_became_active":"2024-03-18T14:42:56.318483+0000","last_became_peered":"2024-03-18T14:42:56.318483+0000","last_unstale":"2024-03-18T14:42:58.319907+0000","last_undegraded":"2024-03-18T14:42:58.319907+0000","last_fullsized":"2024-03-18T14:42:58.319907+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:06.244996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.329040+0000","last_change":"2024-03-18T14:42:57.317184+0000","last_active":"2024-03-18T14:42:57.329040+0000","last_peered":"2024-03-18T14:42:57.329040+0000","last_clean":"2024-03-18T14:42:57.329040+0000","last_became_active":"2024-03-18T14:42:57.316925+0000","last_became_peered":"2024-03-18T14:42:57.316925+0000","last_unstale":"2024-03-18T14:42:57.329040+0000","last_undegraded":"2024-03-18T14:42:57.329040+0000","last_fullsized":"2024-03-18T14:42:57.329040+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:36.914241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323390+0000","last_change":"2024-03-18T14:42:56.321703+0000","last_active":"2024-03-18T14:42:58.323390+0000","last_peered":"2024-03-18T14:42:58.323390+0000","last_clean":"2024-03-18T14:42:58.323390+0000","last_became_active":"2024-03-18T14:42:56.321495+0000","last_became_peered":"2024-03-18T14:42:56.321495+0000","last_unstale":"2024-03-18T14:42:58.323390+0000","last_undegraded":"2024-03-18T14:42:58.323390+0000","last_fullsized":"2024-03-18T14:42:58.323390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:14.308814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334978+0000","last_change":"2024-03-18T14:42:57.329512+0000","last_active":"2024-03-18T14:42:57.334978+0000","last_peered":"2024-03-18T14:42:57.334978+0000","last_clean":"2024-03-18T14:42:57.334978+0000","last_became_active":"2024-03-18T14:42:57.328808+0000","last_became_peered":"2024-03-18T14:42:57.328808+0000","last_unstale":"2024-03-18T14:42:57.334978+0000","last_undegraded":"2024-03-18T14:42:57.334978+0000","last_fullsized":"2024-03-18T14:42:57.334978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:55.847343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324370+0000","last_change":"2024-03-18T14:42:56.315722+0000","last_active":"2024-03-18T14:42:58.324370+0000","last_peered":"2024-03-18T14:42:58.324370+0000","last_clean":"2024-03-18T14:42:58.324370+0000","last_became_active":"2024-03-18T14:42:56.315354+0000","last_became_peered":"2024-03-18T14:42:56.315354+0000","last_unstale":"2024-03-18T14:42:58.324370+0000","last_undegraded":"2024-03-18T14:42:58.324370+0000","last_fullsized":"2024-03-18T14:42:58.324370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:54.845213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333162+0000","last_change":"2024-03-18T14:42:57.325238+0000","last_active":"2024-03-18T14:42:57.333162+0000","last_peered":"2024-03-18T14:42:57.333162+0000","last_clean":"2024-03-18T14:42:57.333162+0000","last_became_active":"2024-03-18T14:42:57.324991+0000","last_became_peered":"2024-03-18T14:42:57.324991+0000","last_unstale":"2024-03-18T14:42:57.333162+0000","last_undegraded":"2024-03-18T14:42:57.333162+0000","last_fullsized":"2024-03-18T14:42:57.333162+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:15.310943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319762+0000","last_change":"2024-03-18T14:42:56.323408+0000","last_active":"2024-03-18T14:42:58.319762+0000","last_peered":"2024-03-18T14:42:58.319762+0000","last_clean":"2024-03-18T14:42:58.319762+0000","last_became_active":"2024-03-18T14:42:56.323126+0000","last_became_peered":"2024-03-18T14:42:56.323126+0000","last_unstale":"2024-03-18T14:42:58.319762+0000","last_undegraded":"2024-03-18T14:42:58.319762+0000","last_fullsized":"2024-03-18T14:42:58.319762+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:10.938463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334857+0000","last_change":"2024-03-18T14:42:57.325813+0000","last_active":"2024-03-18T14:42:57.334857+0000","last_peered":"2024-03-18T14:42:57.334857+0000","last_clean":"2024-03-18T14:42:57.334857+0000","last_became_active":"2024-03-18T14:42:57.325517+0000","last_became_peered":"2024-03-18T14:42:57.325517+0000","last_unstale":"2024-03-18T14:42:57.334857+0000","last_undegraded":"2024-03-18T14:42:57.334857+0000","last_fullsized":"2024-03-18T14:42:57.334857+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:19.877972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319697+0000","last_change":"2024-03-18T14:42:56.321185+0000","last_active":"2024-03-18T14:42:58.319697+0000","last_peered":"2024-03-18T14:42:58.319697+0000","last_clean":"2024-03-18T14:42:58.319697+0000","last_became_active":"2024-03-18T14:42:56.320895+0000","last_became_peered":"2024-03-18T14:42:56.320895+0000","last_unstale":"2024-03-18T14:42:58.319697+0000","last_undegraded":"2024-03-18T14:42:58.319697+0000","last_fullsized":"2024-03-18T14:42:58.319697+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:18.964516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319471+0000","last_change":"2024-03-18T14:42:57.333088+0000","last_active":"2024-03-18T14:42:58.319471+0000","last_peered":"2024-03-18T14:42:58.319471+0000","last_clean":"2024-03-18T14:42:58.319471+0000","last_became_active":"2024-03-18T14:42:57.332789+0000","last_became_peered":"2024-03-18T14:42:57.332789+0000","last_unstale":"2024-03-18T14:42:58.319471+0000","last_undegraded":"2024-03-18T14:42:58.319471+0000","last_fullsized":"2024-03-18T14:42:58.319471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:33:17.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.315285+0000","last_change":"2024-03-18T14:42:56.318459+0000","last_active":"2024-03-18T14:42:57.315285+0000","last_peered":"2024-03-18T14:42:57.315285+0000","last_clean":"2024-03-18T14:42:57.315285+0000","last_became_active":"2024-03-18T14:42:56.318100+0000","last_became_peered":"2024-03-18T14:42:56.318100+0000","last_unstale":"2024-03-18T14:42:57.315285+0000","last_undegraded":"2024-03-18T14:42:57.315285+0000","last_fullsized":"2024-03-18T14:42:57.315285+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:12.140291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.322115+0000","last_change":"2024-03-18T14:42:57.313551+0000","last_active":"2024-03-18T14:42:57.322115+0000","last_peered":"2024-03-18T14:42:57.322115+0000","last_clean":"2024-03-18T14:42:57.322115+0000","last_became_active":"2024-03-18T14:42:57.313303+0000","last_became_peered":"2024-03-18T14:42:57.313303+0000","last_unstale":"2024-03-18T14:42:57.322115+0000","last_undegraded":"2024-03-18T14:42:57.322115+0000","last_fullsized":"2024-03-18T14:42:57.322115+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.322980+0000","last_change":"2024-03-18T14:42:56.317018+0000","last_active":"2024-03-18T14:42:58.322980+0000","last_peered":"2024-03-18T14:42:58.322980+0000","last_clean":"2024-03-18T14:42:58.322980+0000","last_became_active":"2024-03-18T14:42:56.316710+0000","last_became_peered":"2024-03-18T14:42:56.316710+0000","last_unstale":"2024-03-18T14:42:58.322980+0000","last_undegraded":"2024-03-18T14:42:58.322980+0000","last_fullsized":"2024-03-18T14:42:58.322980+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:44.590323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319596+0000","last_change":"2024-03-18T14:42:57.332643+0000","last_active":"2024-03-18T14:42:58.319596+0000","last_peered":"2024-03-18T14:42:58.319596+0000","last_clean":"2024-03-18T14:42:58.319596+0000","last_became_active":"2024-03-18T14:42:57.332353+0000","last_became_peered":"2024-03-18T14:42:57.332353+0000","last_unstale":"2024-03-18T14:42:58.319596+0000","last_undegraded":"2024-03-18T14:42:58.319596+0000","last_fullsized":"2024-03-18T14:42:58.319596+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:11.493713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.339419+0000","last_change":"2024-03-18T14:42:56.323694+0000","last_active":"2024-03-18T14:42:57.339419+0000","last_peered":"2024-03-18T14:42:57.339419+0000","last_clean":"2024-03-18T14:42:57.339419+0000","last_became_active":"2024-03-18T14:42:56.323399+0000","last_became_peered":"2024-03-18T14:42:56.323399+0000","last_unstale":"2024-03-18T14:42:57.339419+0000","last_undegraded":"2024-03-18T14:42:57.339419+0000","last_fullsized":"2024-03-18T14:42:57.339419+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:12.140291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319745+0000","last_change":"2024-03-18T14:42:57.328096+0000","last_active":"2024-03-18T14:42:58.319745+0000","last_peered":"2024-03-18T14:42:58.319745+0000","last_clean":"2024-03-18T14:42:58.319745+0000","last_became_active":"2024-03-18T14:42:57.327794+0000","last_became_peered":"2024-03-18T14:42:57.327794+0000","last_unstale":"2024-03-18T14:42:58.319745+0000","last_undegraded":"2024-03-18T14:42:58.319745+0000","last_fullsized":"2024-03-18T14:42:58.319745+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:53.015812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323948+0000","last_change":"2024-03-18T14:42:56.313601+0000","last_active":"2024-03-18T14:42:58.323948+0000","last_peered":"2024-03-18T14:42:58.323948+0000","last_clean":"2024-03-18T14:42:58.323948+0000","last_became_active":"2024-03-18T14:42:56.313307+0000","last_became_peered":"2024-03-18T14:42:56.313307+0000","last_unstale":"2024-03-18T14:42:58.323948+0000","last_undegraded":"2024-03-18T14:42:58.323948+0000","last_fullsized":"2024-03-18T14:42:58.323948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:35.912112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324487+0000","last_change":"2024-03-18T14:42:57.319047+0000","last_active":"2024-03-18T14:42:58.324487+0000","last_peered":"2024-03-18T14:42:58.324487+0000","last_clean":"2024-03-18T14:42:58.324487+0000","last_became_active":"2024-03-18T14:42:57.318766+0000","last_became_peered":"2024-03-18T14:42:57.318766+0000","last_unstale":"2024-03-18T14:42:58.324487+0000","last_undegraded":"2024-03-18T14:42:58.324487+0000","last_fullsized":"2024-03-18T14:42:58.324487+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:07.997004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333654+0000","last_change":"2024-03-18T14:42:56.319098+0000","last_active":"2024-03-18T14:42:57.333654+0000","last_peered":"2024-03-18T14:42:57.333654+0000","last_clean":"2024-03-18T14:42:57.333654+0000","last_became_active":"2024-03-18T14:42:56.318482+0000","last_became_peered":"2024-03-18T14:42:56.318482+0000","last_unstale":"2024-03-18T14:42:57.333654+0000","last_undegraded":"2024-03-18T14:42:57.333654+0000","last_fullsized":"2024-03-18T14:42:57.333654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:53:18.875842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332872+0000","last_change":"2024-03-18T14:42:57.319520+0000","last_active":"2024-03-18T14:42:57.332872+0000","last_peered":"2024-03-18T14:42:57.332872+0000","last_clean":"2024-03-18T14:42:57.332872+0000","last_became_active":"2024-03-18T14:42:57.319194+0000","last_became_peered":"2024-03-18T14:42:57.319194+0000","last_unstale":"2024-03-18T14:42:57.332872+0000","last_undegraded":"2024-03-18T14:42:57.332872+0000","last_fullsized":"2024-03-18T14:42:57.332872+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.333350+0000","last_change":"2024-03-18T14:42:56.320173+0000","last_active":"2024-03-18T14:42:57.333350+0000","last_peered":"2024-03-18T14:42:57.333350+0000","last_clean":"2024-03-18T14:42:57.333350+0000","last_became_active":"2024-03-18T14:42:56.319845+0000","last_became_peered":"2024-03-18T14:42:56.319845+0000","last_unstale":"2024-03-18T14:42:57.333350+0000","last_undegraded":"2024-03-18T14:42:57.333350+0000","last_fullsized":"2024-03-18T14:42:57.333350+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:10:30.596234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.323490+0000","last_change":"2024-03-18T14:42:57.320961+0000","last_active":"2024-03-18T14:42:57.323490+0000","last_peered":"2024-03-18T14:42:57.323490+0000","last_clean":"2024-03-18T14:42:57.323490+0000","last_became_active":"2024-03-18T14:42:57.320679+0000","last_became_peered":"2024-03-18T14:42:57.320679+0000","last_unstale":"2024-03-18T14:42:57.323490+0000","last_undegraded":"2024-03-18T14:42:57.323490+0000","last_fullsized":"2024-03-18T14:42:57.323490+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:22.091349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.313155+0000","last_change":"2024-03-18T14:42:56.317130+0000","last_active":"2024-03-18T14:42:57.313155+0000","last_peered":"2024-03-18T14:42:57.313155+0000","last_clean":"2024-03-18T14:42:57.313155+0000","last_became_active":"2024-03-18T14:42:56.316836+0000","last_became_peered":"2024-03-18T14:42:56.316836+0000","last_unstale":"2024-03-18T14:42:57.313155+0000","last_undegraded":"2024-03-18T14:42:57.313155+0000","last_fullsized":"2024-03-18T14:42:57.313155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:12.291125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.326415+0000","last_change":"2024-03-18T14:42:57.317216+0000","last_active":"2024-03-18T14:42:57.326415+0000","last_peered":"2024-03-18T14:42:57.326415+0000","last_clean":"2024-03-18T14:42:57.326415+0000","last_became_active":"2024-03-18T14:42:57.316940+0000","last_became_peered":"2024-03-18T14:42:57.316940+0000","last_unstale":"2024-03-18T14:42:57.326415+0000","last_undegraded":"2024-03-18T14:42:57.326415+0000","last_fullsized":"2024-03-18T14:42:57.326415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:37.127269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.335432+0000","last_change":"2024-03-18T14:42:56.320581+0000","last_active":"2024-03-18T14:42:57.335432+0000","last_peered":"2024-03-18T14:42:57.335432+0000","last_clean":"2024-03-18T14:42:57.335432+0000","last_became_active":"2024-03-18T14:42:56.319720+0000","last_became_peered":"2024-03-18T14:42:56.319720+0000","last_unstale":"2024-03-18T14:42:57.335432+0000","last_undegraded":"2024-03-18T14:42:57.335432+0000","last_fullsized":"2024-03-18T14:42:57.335432+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:03.169629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334998+0000","last_change":"2024-03-18T14:42:57.329810+0000","last_active":"2024-03-18T14:42:57.334998+0000","last_peered":"2024-03-18T14:42:57.334998+0000","last_clean":"2024-03-18T14:42:57.334998+0000","last_became_active":"2024-03-18T14:42:57.329517+0000","last_became_peered":"2024-03-18T14:42:57.329517+0000","last_unstale":"2024-03-18T14:42:57.334998+0000","last_undegraded":"2024-03-18T14:42:57.334998+0000","last_fullsized":"2024-03-18T14:42:57.334998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:05:36.914241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.305706+0000","last_change":"2024-03-18T14:42:56.314280+0000","last_active":"2024-03-18T14:42:57.305706+0000","last_peered":"2024-03-18T14:42:57.305706+0000","last_clean":"2024-03-18T14:42:57.305706+0000","last_became_active":"2024-03-18T14:42:56.313977+0000","last_became_peered":"2024-03-18T14:42:56.313977+0000","last_unstale":"2024-03-18T14:42:57.305706+0000","last_undegraded":"2024-03-18T14:42:57.305706+0000","last_fullsized":"2024-03-18T14:42:57.305706+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:11:55.790619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324123+0000","last_change":"2024-03-18T14:42:57.327062+0000","last_active":"2024-03-18T14:42:58.324123+0000","last_peered":"2024-03-18T14:42:58.324123+0000","last_clean":"2024-03-18T14:42:58.324123+0000","last_became_active":"2024-03-18T14:42:57.326696+0000","last_became_peered":"2024-03-18T14:42:57.326696+0000","last_unstale":"2024-03-18T14:42:58.324123+0000","last_undegraded":"2024-03-18T14:42:58.324123+0000","last_fullsized":"2024-03-18T14:42:58.324123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:03.049979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.307611+0000","last_change":"2024-03-18T14:42:56.323171+0000","last_active":"2024-03-18T14:42:57.307611+0000","last_peered":"2024-03-18T14:42:57.307611+0000","last_clean":"2024-03-18T14:42:57.307611+0000","last_became_active":"2024-03-18T14:42:56.322898+0000","last_became_peered":"2024-03-18T14:42:56.322898+0000","last_unstale":"2024-03-18T14:42:57.307611+0000","last_undegraded":"2024-03-18T14:42:57.307611+0000","last_fullsized":"2024-03-18T14:42:57.307611+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:31.020377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319645+0000","last_change":"2024-03-18T14:42:57.327604+0000","last_active":"2024-03-18T14:42:58.319645+0000","last_peered":"2024-03-18T14:42:58.319645+0000","last_clean":"2024-03-18T14:42:58.319645+0000","last_became_active":"2024-03-18T14:42:57.327196+0000","last_became_peered":"2024-03-18T14:42:57.327196+0000","last_unstale":"2024-03-18T14:42:58.319645+0000","last_undegraded":"2024-03-18T14:42:58.319645+0000","last_fullsized":"2024-03-18T14:42:58.319645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:36.168961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.305414+0000","last_change":"2024-03-18T14:42:56.319304+0000","last_active":"2024-03-18T14:42:57.305414+0000","last_peered":"2024-03-18T14:42:57.305414+0000","last_clean":"2024-03-18T14:42:57.305414+0000","last_became_active":"2024-03-18T14:42:56.318996+0000","last_became_peered":"2024-03-18T14:42:56.318996+0000","last_unstale":"2024-03-18T14:42:57.305414+0000","last_undegraded":"2024-03-18T14:42:57.305414+0000","last_fullsized":"2024-03-18T14:42:57.305414+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:50.599974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.330499+0000","last_change":"2024-03-18T14:42:57.327098+0000","last_active":"2024-03-18T14:42:57.330499+0000","last_peered":"2024-03-18T14:42:57.330499+0000","last_clean":"2024-03-18T14:42:57.330499+0000","last_became_active":"2024-03-18T14:42:57.326829+0000","last_became_peered":"2024-03-18T14:42:57.326829+0000","last_unstale":"2024-03-18T14:42:57.330499+0000","last_undegraded":"2024-03-18T14:42:57.330499+0000","last_fullsized":"2024-03-18T14:42:57.330499+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:32.022506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324139+0000","last_change":"2024-03-18T14:42:56.315773+0000","last_active":"2024-03-18T14:42:58.324139+0000","last_peered":"2024-03-18T14:42:58.324139+0000","last_clean":"2024-03-18T14:42:58.324139+0000","last_became_active":"2024-03-18T14:42:56.315441+0000","last_became_peered":"2024-03-18T14:42:56.315441+0000","last_unstale":"2024-03-18T14:42:58.324139+0000","last_undegraded":"2024-03-18T14:42:58.324139+0000","last_fullsized":"2024-03-18T14:42:58.324139+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:36.125139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.329061+0000","last_change":"2024-03-18T14:42:57.317878+0000","last_active":"2024-03-18T14:42:57.329061+0000","last_peered":"2024-03-18T14:42:57.329061+0000","last_clean":"2024-03-18T14:42:57.329061+0000","last_became_active":"2024-03-18T14:42:57.317587+0000","last_became_peered":"2024-03-18T14:42:57.317587+0000","last_unstale":"2024-03-18T14:42:57.329061+0000","last_undegraded":"2024-03-18T14:42:57.329061+0000","last_fullsized":"2024-03-18T14:42:57.329061+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:19.633689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.305522+0000","last_change":"2024-03-18T14:42:56.309300+0000","last_active":"2024-03-18T14:42:57.305522+0000","last_peered":"2024-03-18T14:42:57.305522+0000","last_clean":"2024-03-18T14:42:57.305522+0000","last_became_active":"2024-03-18T14:42:56.309031+0000","last_became_peered":"2024-03-18T14:42:56.309031+0000","last_unstale":"2024-03-18T14:42:57.305522+0000","last_undegraded":"2024-03-18T14:42:57.305522+0000","last_fullsized":"2024-03-18T14:42:57.305522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:51:26.593309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319680+0000","last_change":"2024-03-18T14:42:57.323859+0000","last_active":"2024-03-18T14:42:58.319680+0000","last_peered":"2024-03-18T14:42:58.319680+0000","last_clean":"2024-03-18T14:42:58.319680+0000","last_became_active":"2024-03-18T14:42:57.323530+0000","last_became_peered":"2024-03-18T14:42:57.323530+0000","last_unstale":"2024-03-18T14:42:58.319680+0000","last_undegraded":"2024-03-18T14:42:58.319680+0000","last_fullsized":"2024-03-18T14:42:58.319680+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:07.997004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.319694+0000","last_change":"2024-03-18T14:42:56.322152+0000","last_active":"2024-03-18T14:42:57.319694+0000","last_peered":"2024-03-18T14:42:57.319694+0000","last_clean":"2024-03-18T14:42:57.319694+0000","last_became_active":"2024-03-18T14:42:56.321863+0000","last_became_peered":"2024-03-18T14:42:56.321863+0000","last_unstale":"2024-03-18T14:42:57.319694+0000","last_undegraded":"2024-03-18T14:42:57.319694+0000","last_fullsized":"2024-03-18T14:42:57.319694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:51.796731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.334878+0000","last_change":"2024-03-18T14:42:57.328288+0000","last_active":"2024-03-18T14:42:57.334878+0000","last_peered":"2024-03-18T14:42:57.334878+0000","last_clean":"2024-03-18T14:42:57.334878+0000","last_became_active":"2024-03-18T14:42:57.328016+0000","last_became_peered":"2024-03-18T14:42:57.328016+0000","last_unstale":"2024-03-18T14:42:57.334878+0000","last_undegraded":"2024-03-18T14:42:57.334878+0000","last_fullsized":"2024-03-18T14:42:57.334878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.318645+0000","last_change":"2024-03-18T14:42:56.321847+0000","last_active":"2024-03-18T14:42:57.318645+0000","last_peered":"2024-03-18T14:42:57.318645+0000","last_clean":"2024-03-18T14:42:57.318645+0000","last_became_active":"2024-03-18T14:42:56.321532+0000","last_became_peered":"2024-03-18T14:42:56.321532+0000","last_unstale":"2024-03-18T14:42:57.318645+0000","last_undegraded":"2024-03-18T14:42:57.318645+0000","last_fullsized":"2024-03-18T14:42:57.318645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:12.291125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332633+0000","last_change":"2024-03-18T14:42:57.331206+0000","last_active":"2024-03-18T14:42:57.332633+0000","last_peered":"2024-03-18T14:42:57.332633+0000","last_clean":"2024-03-18T14:42:57.332633+0000","last_became_active":"2024-03-18T14:42:57.330863+0000","last_became_peered":"2024-03-18T14:42:57.330863+0000","last_unstale":"2024-03-18T14:42:57.332633+0000","last_undegraded":"2024-03-18T14:42:57.332633+0000","last_fullsized":"2024-03-18T14:42:57.332633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:26.896193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.323797+0000","last_change":"2024-03-18T14:42:56.319526+0000","last_active":"2024-03-18T14:42:58.323797+0000","last_peered":"2024-03-18T14:42:58.323797+0000","last_clean":"2024-03-18T14:42:58.323797+0000","last_became_active":"2024-03-18T14:42:56.318876+0000","last_became_peered":"2024-03-18T14:42:56.318876+0000","last_unstale":"2024-03-18T14:42:58.323797+0000","last_undegraded":"2024-03-18T14:42:58.323797+0000","last_fullsized":"2024-03-18T14:42:58.323797+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:00:54.261893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.332957+0000","last_change":"2024-03-18T14:42:57.320372+0000","last_active":"2024-03-18T14:42:57.332957+0000","last_peered":"2024-03-18T14:42:57.332957+0000","last_clean":"2024-03-18T14:42:57.332957+0000","last_became_active":"2024-03-18T14:42:57.320097+0000","last_became_peered":"2024-03-18T14:42:57.320097+0000","last_unstale":"2024-03-18T14:42:57.332957+0000","last_undegraded":"2024-03-18T14:42:57.332957+0000","last_fullsized":"2024-03-18T14:42:57.332957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:22.950609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.313943+0000","last_change":"2024-03-18T14:42:56.323658+0000","last_active":"2024-03-18T14:42:57.313943+0000","last_peered":"2024-03-18T14:42:57.313943+0000","last_clean":"2024-03-18T14:42:57.313943+0000","last_became_active":"2024-03-18T14:42:56.323411+0000","last_became_peered":"2024-03-18T14:42:56.323411+0000","last_unstale":"2024-03-18T14:42:57.313943+0000","last_undegraded":"2024-03-18T14:42:57.313943+0000","last_fullsized":"2024-03-18T14:42:57.313943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:44:12.140291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.328884+0000","last_change":"2024-03-18T14:42:57.319067+0000","last_active":"2024-03-18T14:42:57.328884+0000","last_peered":"2024-03-18T14:42:57.328884+0000","last_clean":"2024-03-18T14:42:57.328884+0000","last_became_active":"2024-03-18T14:42:57.318817+0000","last_became_peered":"2024-03-18T14:42:57.318817+0000","last_unstale":"2024-03-18T14:42:57.328884+0000","last_undegraded":"2024-03-18T14:42:57.328884+0000","last_fullsized":"2024-03-18T14:42:57.328884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:11.940592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.319402+0000","last_change":"2024-03-18T14:42:56.315423+0000","last_active":"2024-03-18T14:42:58.319402+0000","last_peered":"2024-03-18T14:42:58.319402+0000","last_clean":"2024-03-18T14:42:58.319402+0000","last_became_active":"2024-03-18T14:42:56.314846+0000","last_became_peered":"2024-03-18T14:42:56.314846+0000","last_unstale":"2024-03-18T14:42:58.319402+0000","last_undegraded":"2024-03-18T14:42:58.319402+0000","last_fullsized":"2024-03-18T14:42:58.319402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:28.354432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T14:42:58.324041+0000","last_change":"2024-03-18T14:42:57.322324+0000","last_active":"2024-03-18T14:42:58.324041+0000","last_peered":"2024-03-18T14:42:58.324041+0000","last_clean":"2024-03-18T14:42:58.324041+0000","last_became_active":"2024-03-18T14:42:57.322044+0000","last_became_peered":"2024-03-18T14:42:57.322044+0000","last_unstale":"2024-03-18T14:42:58.324041+0000","last_undegraded":"2024-03-18T14:42:58.324041+0000","last_fullsized":"2024-03-18T14:42:58.324041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:56.295570+0000","last_clean_scrub_stamp":"2024-03-18T14:42:56.295570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:52:36.862870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.308250+0000","last_change":"2024-03-18T14:42:56.324502+0000","last_active":"2024-03-18T14:42:57.308250+0000","last_peered":"2024-03-18T14:42:57.308250+0000","last_clean":"2024-03-18T14:42:57.308250+0000","last_became_active":"2024-03-18T14:42:56.324303+0000","last_became_peered":"2024-03-18T14:42:56.324303+0000","last_unstale":"2024-03-18T14:42:57.308250+0000","last_undegraded":"2024-03-18T14:42:57.308250+0000","last_fullsized":"2024-03-18T14:42:57.308250+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:42:55.293440+0000","last_clean_scrub_stamp":"2024-03-18T14:42:55.293440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:21.948479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T14:42:57.306422+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:42:57.306422+0000","last_peered":"2024-03-18T14:42:57.306422+0000","last_clean":"2024-03-18T14:42:57.306422+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:42:57.306422+0000","last_undegraded":"2024-03-18T14:42:57.306422+0000","last_fullsized":"2024-03-18T14:42:57.306422+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":780065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":189661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1036288,"data_stored":778417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:43:00.674 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T14:43:00.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T14:43:00.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.819+0000 7f6345c44640 1 Processor -- start 2024-03-18T14:43:00.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.819+0000 7f6345c44640 1 -- start start 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f6345c44640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f63400ce930 0x7f634006cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f6345c44640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 0x7f63400d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f6345c44640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63400d0f30 con 0x7f63400d0680 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f6345c44640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f634006d480 con 0x7f63400ce930 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f6345c44640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f63400b1070 con 0x7f634006d020 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63400d0680 0x7f634015d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41634/0 (socket says 172.21.15.6:41634) 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633ffff640 1 -- 172.21.15.6:0/983178449 learned_addr learned my addr 172.21.15.6:0/983178449 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:00.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633effd640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 0x7f63400d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633f7fe640 1 --2- 172.21.15.6:0/983178449 >> v2:172.21.15.92:3300/0 conn(0x7f63400ce930 0x7f634006cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903807865 0 0) 0x7f63400d0f30 con 0x7f63400d0680 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6328003520 con 0x7f63400d0680 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2892249170 0 0) 0x7f6328003520 con 0x7f63400d0680 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 msgr2=0x7f63400d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 0x7f63400d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 >> v2:172.21.15.92:3300/0 conn(0x7f63400ce930 msgr2=0x7f634006cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 --2- 172.21.15.6:0/983178449 >> v2:172.21.15.92:3300/0 conn(0x7f63400ce930 0x7f634006cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63400b1350 con 0x7f63400d0680 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f63380019f0 con 0x7f63400d0680 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633effd640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 0x7f63400d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.820+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 >> v1:172.21.15.6:6789/0 conn(0x7f63400d0680 legacy=0x7f634015d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633e7fc640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63280039b0 0x7f6328003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633e7fc640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328004800 0x7f6328004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633e7fc640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 0x7f63280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f63400b1070 con 0x7f63280039b0 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f632800b9c0 con 0x7f6328004800 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f632800bb40 con 0x7f634006d020 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633f7fe640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63280039b0 0x7f6328003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633ffff640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 0x7f63280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633effd640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328004800 0x7f6328004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633f7fe640 1 -- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 msgr2=0x7f63280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633f7fe640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634006d020 0x7f63280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633f7fe640 1 -- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328004800 msgr2=0x7f6328004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633f7fe640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328004800 0x7f6328004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.821+0000 7f633f7fe640 1 -- 172.21.15.6:0/983178449 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6328003520 con 0x7f63280039b0 2024-03-18T14:43:00.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.822+0000 7f633f7fe640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63280039b0 0x7f6328003dd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6330009ab0 tx=0x7f6328005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e66bace7b80385 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.822+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6330009680 con 0x7f63280039b0 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.822+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6330005990 con 0x7f63280039b0 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.822+0000 7f633e7fc640 1 -- 172.21.15.6:0/983178449 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6330005d80 con 0x7f63280039b0 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.822+0000 7f6345c44640 1 -- 172.21.15.6:0/983178449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63280039b0 msgr2=0x7f6328003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.822+0000 7f6345c44640 1 --2- 172.21.15.6:0/983178449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63280039b0 0x7f6328003dd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6330009ab0 tx=0x7f6328005830 comp rx=0 tx=0).stop 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.822+0000 7f633ffff640 1 -- 172.21.15.6:0/983178449 reap_dead start 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.823+0000 7f6345c44640 1 -- 172.21.15.6:0/983178449 shutdown_connections 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.823+0000 7f6345c44640 1 -- 172.21.15.6:0/983178449 >> 172.21.15.6:0/983178449 conn(0x7f634008b0c0 msgr2=0x7f63400d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.823+0000 7f6345c44640 1 -- 172.21.15.6:0/983178449 shutdown_connections 2024-03-18T14:43:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.823+0000 7f6345c44640 1 -- 172.21.15.6:0/983178449 wait complete. 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.824+0000 7f6345c44640 1 Processor -- start 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.824+0000 7f6345c44640 1 -- start start 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63280039b0 0x7f6340156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400b4630 0x7f63400b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400b4ff0 0x7f63401f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f63400cee90 con 0x7f63400b4630 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f63400ced10 con 0x7f63280039b0 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f63400d0140 con 0x7f63400b4ff0 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400b4630 0x7f63400b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400b4630 0x7f63400b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58038/0 (socket says 172.21.15.6:58038) 2024-03-18T14:43:00.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 -- 172.21.15.6:0/4147671716 learned_addr learned my addr 172.21.15.6:0/4147671716 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633f7fe640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63280039b0 0x7f6340156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633ffff640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400b4ff0 0x7f63401f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 -- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400b4ff0 msgr2=0x7f63401f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400b4ff0 0x7f63401f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 -- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63280039b0 msgr2=0x7f6340156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63280039b0 0x7f6340156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 -- 172.21.15.6:0/4147671716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63400b5470 con 0x7f63400b4630 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633ffff640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400b4ff0 0x7f63401f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633f7fe640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63280039b0 0x7f6340156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f633effd640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400b4630 0x7f63400b4ab0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6334011960 tx=0x7f6334011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6327fff640 1 -- 172.21.15.6:0/4147671716 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633400faf0 con 0x7f63400b4630 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6327fff640 1 -- 172.21.15.6:0/4147671716 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633400fc90 con 0x7f63400b4630 2024-03-18T14:43:00.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6327fff640 1 -- 172.21.15.6:0/4147671716 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6334004030 con 0x7f63400b4630 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63401f4080 con 0x7f63400b4630 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.825+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63401f4600 con 0x7f63400b4630 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.826+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63400751b0 con 0x7f63400b4630 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.827+0000 7f6327fff640 1 -- 172.21.15.6:0/4147671716 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6334012040 con 0x7f63400b4630 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.827+0000 7f6327fff640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632c06bbd0 0x7f632c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.827+0000 7f6327fff640 1 -- 172.21.15.6:0/4147671716 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f633408af90 con 0x7f63400b4630 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.828+0000 7f633f7fe640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632c06bbd0 0x7f632c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:00.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.828+0000 7f633f7fe640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632c06bbd0 0x7f632c06e090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6330035000 tx=0x7f6330030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:00.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:00.832+0000 7f6327fff640 1 -- 172.21.15.6:0/4147671716 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6334019070 con 0x7f63400b4630 2024-03-18T14:43:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.105+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f634006dad0 con 0x7f63400b4630 2024-03-18T14:43:01.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.107+0000 7f6327fff640 1 -- 172.21.15.6:0/4147671716 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f63340598f0 con 0x7f63400b4630 2024-03-18T14:43:01.109 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:43:01.109 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:43:01.109 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-18T14:43:01.109 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:43:01.109 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-18T14:43:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:43:01.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632c06bbd0 msgr2=0x7f632c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632c06bbd0 0x7f632c06e090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6330035000 tx=0x7f6330030000 comp rx=0 tx=0).stop 2024-03-18T14:43:01.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400b4630 msgr2=0x7f63400b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400b4630 0x7f63400b4ab0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6334011960 tx=0x7f6334011e20 comp rx=0 tx=0).stop 2024-03-18T14:43:01.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 shutdown_connections 2024-03-18T14:43:01.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632c06bbd0 0x7f632c06e090 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400b4ff0 0x7f63401f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400b4630 0x7f63400b4ab0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 --2- 172.21.15.6:0/4147671716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63280039b0 0x7f6340156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.109+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 >> 172.21.15.6:0/4147671716 conn(0x7f634008b0c0 msgr2=0x7f63400d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.110+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 shutdown_connections 2024-03-18T14:43:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.110+0000 7f6345c44640 1 -- 172.21.15.6:0/4147671716 wait complete. 2024-03-18T14:43:01.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:01.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.250+0000 7fbef68f1640 1 Processor -- start 2024-03-18T14:43:01.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.250+0000 7fbef68f1640 1 -- start start 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbef68f1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbef006cce0 0x7fbef006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbef68f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef00b17b0 0x7fbef00b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbef68f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef00cfd80 con 0x7fbef00ce3c0 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbef68f1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbef00b2df0 con 0x7fbef006cce0 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbef68f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbef006d600 con 0x7fbef00b17b0 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbeeffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbef006cce0 0x7fbef006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbef4e8e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbef00ce3c0 0x7fbef00ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41646/0 (socket says 172.21.15.6:41646) 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbef4e8e640 1 -- 172.21.15.6:0/2927380632 learned_addr learned my addr 172.21.15.6:0/2927380632 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbeef7fe640 1 --2- 172.21.15.6:0/2927380632 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef00b17b0 0x7fbef00b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbeeffff640 1 -- 172.21.15.6:0/2927380632 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef00b17b0 msgr2=0x7fbef00b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbeeffff640 1 --2- 172.21.15.6:0/2927380632 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef00b17b0 0x7fbef00b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbeeffff640 1 -- 172.21.15.6:0/2927380632 >> v1:172.21.15.6:6789/0 conn(0x7fbef00ce3c0 legacy=0x7fbef00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbeeeffd640 1 -- 172.21.15.6:0/2927380632 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004144926 0 0) 0x7fbef00cfd80 con 0x7fbef00ce3c0 2024-03-18T14:43:01.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.251+0000 7fbeeffff640 1 -- 172.21.15.6:0/2927380632 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef00b2270 con 0x7fbef006cce0 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.252+0000 7fbeeffff640 1 --2- 172.21.15.6:0/2927380632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef006cce0 0x7fbef006d0c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbee40097a0 tx=0x7fbee402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c293ec5d70b941f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.252+0000 7fbeeeffd640 1 -- 172.21.15.6:0/2927380632 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee40315a0 con 0x7fbef006cce0 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.252+0000 7fbeeeffd640 1 -- 172.21.15.6:0/2927380632 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee4037070 con 0x7fbef006cce0 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.252+0000 7fbeeeffd640 1 -- 172.21.15.6:0/2927380632 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4031960 con 0x7fbef006cce0 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 -- 172.21.15.6:0/2927380632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef006cce0 msgr2=0x7fbef006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 --2- 172.21.15.6:0/2927380632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef006cce0 0x7fbef006d0c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbee40097a0 tx=0x7fbee402b840 comp rx=0 tx=0).stop 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 -- 172.21.15.6:0/2927380632 shutdown_connections 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 --2- 172.21.15.6:0/2927380632 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef00b17b0 0x7fbef00b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 --2- 172.21.15.6:0/2927380632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef006cce0 0x7fbef006d0c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 -- 172.21.15.6:0/2927380632 >> 172.21.15.6:0/2927380632 conn(0x7fbef008b130 msgr2=0x7fbef008b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:01.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 -- 172.21.15.6:0/2927380632 shutdown_connections 2024-03-18T14:43:01.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.253+0000 7fbef68f1640 1 -- 172.21.15.6:0/2927380632 wait complete. 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.254+0000 7fbef68f1640 1 Processor -- start 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.254+0000 7fbef68f1640 1 -- start start 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef68f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef006cce0 0x7fbef00b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef68f1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef00b17b0 0x7fbef00b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef68f1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef00ce3c0 0x7fbef00c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef68f1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbef00b2b10 con 0x7fbef00ce3c0 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef68f1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbef00b2830 con 0x7fbef00b17b0 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef68f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbef015e330 con 0x7fbef006cce0 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef00ce3c0 0x7fbef00c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef00ce3c0 0x7fbef00c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58046/0 (socket says 172.21.15.6:58046) 2024-03-18T14:43:01.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 -- 172.21.15.6:0/731481772 learned_addr learned my addr 172.21.15.6:0/731481772 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbeeffff640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef006cce0 0x7fbef00b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbeef7fe640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef00b17b0 0x7fbef00b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 -- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef006cce0 msgr2=0x7fbef00b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef006cce0 0x7fbef00b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 -- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef00b17b0 msgr2=0x7fbef00b62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef00b17b0 0x7fbef00b62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbef4e8e640 1 -- 172.21.15.6:0/731481772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef015f900 con 0x7fbef00ce3c0 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.255+0000 7fbeef7fe640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef00b17b0 0x7fbef00b62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.256+0000 7fbef4e8e640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef00ce3c0 0x7fbef00c78c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c7a0 tx=0x7fbee800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.256+0000 7fbeecff9640 1 -- 172.21.15.6:0/731481772 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8018070 con 0x7fbef00ce3c0 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.256+0000 7fbeecff9640 1 -- 172.21.15.6:0/731481772 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee800a640 con 0x7fbef00ce3c0 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.256+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef015fbe0 con 0x7fbef00ce3c0 2024-03-18T14:43:01.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.256+0000 7fbeecff9640 1 -- 172.21.15.6:0/731481772 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee800aab0 con 0x7fbef00ce3c0 2024-03-18T14:43:01.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.256+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef01601a0 con 0x7fbef00ce3c0 2024-03-18T14:43:01.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.257+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef009fa80 con 0x7fbef00ce3c0 2024-03-18T14:43:01.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.257+0000 7fbeecff9640 1 -- 172.21.15.6:0/731481772 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbee801b070 con 0x7fbef00ce3c0 2024-03-18T14:43:01.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.258+0000 7fbeecff9640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:01.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.258+0000 7fbeecff9640 1 -- 172.21.15.6:0/731481772 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbee808a170 con 0x7fbef00ce3c0 2024-03-18T14:43:01.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.258+0000 7fbeeffff640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:01.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.258+0000 7fbeeffff640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbee4036000 tx=0x7fbee4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:01.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.262+0000 7fbeecff9640 1 -- 172.21.15.6:0/731481772 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbee8058480 con 0x7fbef00ce3c0 2024-03-18T14:43:01.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.483+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbef004f620 con 0x7fbef00ce3c0 2024-03-18T14:43:01.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.483+0000 7fbeecff9640 1 -- 172.21.15.6:0/731481772 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5169 (secure 0 0 0) 0x7fbee805c520 con 0x7fbef00ce3c0 2024-03-18T14:43:01.485 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:01.485 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:42:56.672813+0000","modified":"2024-03-18T14:42:58.672583+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5529":{"gid":5529,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2677248951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2677248951},{"type":"v1","addr":"172.21.15.6:6834","nonce":2677248951}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5529,"qdb_cluster":[5529]},"id":2}]} 2024-03-18T14:43:01.485 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 15 2024-03-18T14:43:01.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.485+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 msgr2=0x7fbedc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.485+0000 7fbef68f1640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbee4036000 tx=0x7fbee4008030 comp rx=0 tx=0).stop 2024-03-18T14:43:01.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.485+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef00ce3c0 msgr2=0x7fbef00c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:01.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.485+0000 7fbef68f1640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef00ce3c0 0x7fbef00c78c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c7a0 tx=0x7fbee800cc60 comp rx=0 tx=0).stop 2024-03-18T14:43:01.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 shutdown_connections 2024-03-18T14:43:01.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbef00ce3c0 0x7fbef00c78c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef00b17b0 0x7fbef00b62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 --2- 172.21.15.6:0/731481772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef006cce0 0x7fbef00b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:01.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 >> 172.21.15.6:0/731481772 conn(0x7fbef008b130 msgr2=0x7fbef00b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:01.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 shutdown_connections 2024-03-18T14:43:01.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:01.486+0000 7fbef68f1640 1 -- 172.21.15.6:0/731481772 wait complete. 2024-03-18T14:43:01.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T14:42:56.672813+0000', 'modified': '2024-03-18T14:42:58.672583+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5529}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5529': {'gid': 5529, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2677248951', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2677248951}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2677248951}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5529, 'qdb_cluster': [5529]} 2024-03-18T14:43:01.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T14:43:01.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T14:43:01.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-18T14:43:01.988+0000 7f2cb0137640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T14:43:02.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.002+0000 7fce025d3640 1 Processor -- start 2024-03-18T14:43:02.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.002+0000 7fce025d3640 1 -- start start 2024-03-18T14:43:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.003+0000 7fce025d3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcdfc0d2180 0x7fcdfc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.003+0000 7fce025d3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdfc0cd450 con 0x7fcdfc15f2a0 2024-03-18T14:43:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.003+0000 7fce025d3640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdfc0cd730 con 0x7fcdfc0ccc80 2024-03-18T14:43:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.003+0000 7fce025d3640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcdfc15f680 con 0x7fcdfc0d2180 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.003+0000 7fcdfbfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcdfc15f2a0 0x7fcdfc0d1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41650/0 (socket says 172.21.15.6:41650) 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.003+0000 7fcdfbfff640 1 -- 172.21.15.6:0/1238669447 learned_addr learned my addr 172.21.15.6:0/1238669447 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/1238669447 >> v2:172.21.15.92:3300/0 conn(0x7fcdfc0d2180 0x7fcdfc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 264759503 0 0) 0x7fcdfc0cd450 con 0x7fcdfc15f2a0 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde4003520 con 0x7fcdfc15f2a0 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 373565143 0 0) 0x7fcde4003520 con 0x7fcdfc15f2a0 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 >> v2:172.21.15.92:3300/0 conn(0x7fcdfc0d2180 msgr2=0x7fcdfc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 --2- 172.21.15.6:0/1238669447 >> v2:172.21.15.92:3300/0 conn(0x7fcdfc0d2180 0x7fcdfc0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 >> v1:172.21.15.92:6789/0 conn(0x7fcdfc0ccc80 legacy=0x7fcdfc0c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdfc0cda10 con 0x7fcdfc15f2a0 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fcdec001670 con 0x7fcdfc15f2a0 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 >> v1:172.21.15.6:6789/0 conn(0x7fcdfc15f2a0 legacy=0x7fcdfc0d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde4003a00 0x7fcde4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc15f680 con 0x7fcde40051c0 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcde400c0f0 con 0x7fcde4004800 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcde400c2f0 con 0x7fcde4003a00 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fce00b70640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde4003a00 0x7fcde4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fce00b70640 1 -- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde4003a00 msgr2=0x7fcde4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fce00b70640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde4003a00 0x7fcde4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fce00b70640 1 -- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcde4004800 msgr2=0x7fcde4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fce00b70640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcde4004800 0x7fcde4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.004+0000 7fce00b70640 1 -- 172.21.15.6:0/1238669447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde4003520 con 0x7fcde40051c0 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.005+0000 7fce00b70640 1 -- 172.21.15.6:0/1238669447 reap_dead start 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.005+0000 7fce00b70640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4001e50 tx=0x7fcde4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c651809d08f9aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4001a70 con 0x7fcde40051c0 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf4003bb0 con 0x7fcde40051c0 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fcdfaffd640 1 -- 172.21.15.6:0/1238669447 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40050d0 con 0x7fcde40051c0 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fce025d3640 1 -- 172.21.15.6:0/1238669447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde40051c0 msgr2=0x7fcde400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fce025d3640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4001e50 tx=0x7fcde4005830 comp rx=0 tx=0).stop 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fce025d3640 1 -- 172.21.15.6:0/1238669447 shutdown_connections 2024-03-18T14:43:02.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fce025d3640 1 --2- 172.21.15.6:0/1238669447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcde40051c0 0x7fcde400b9c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fce025d3640 1 -- 172.21.15.6:0/1238669447 >> 172.21.15.6:0/1238669447 conn(0x7fcdfc09ae00 msgr2=0x7fcdfc08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fce025d3640 1 -- 172.21.15.6:0/1238669447 shutdown_connections 2024-03-18T14:43:02.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.006+0000 7fce025d3640 1 -- 172.21.15.6:0/1238669447 wait complete. 2024-03-18T14:43:02.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.008+0000 7fce025d3640 1 Processor -- start 2024-03-18T14:43:02.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.008+0000 7fce025d3640 1 -- start start 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.009+0000 7fce025d3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde40051c0 0x7fcdfc1f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.009+0000 7fce025d3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b6d50 0x7fcdfc1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.009+0000 7fce025d3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdfc0b71d0 0x7fcdfc15b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.009+0000 7fce025d3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc0cd0f0 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.009+0000 7fce025d3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc0cdfd0 con 0x7fcdfc0b71d0 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.009+0000 7fce025d3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcdfc0cc740 con 0x7fcde40051c0 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b6d50 0x7fcdfc1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b6d50 0x7fcdfc1f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58068/0 (socket says 172.21.15.6:58068) 2024-03-18T14:43:02.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 -- 172.21.15.6:0/1703235283 learned_addr learned my addr 172.21.15.6:0/1703235283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde40051c0 0x7fcdfc1f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fce00b70640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdfc0b71d0 0x7fcdfc15b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 -- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde40051c0 msgr2=0x7fcdfc1f7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde40051c0 0x7fcdfc1f7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 -- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdfc0b71d0 msgr2=0x7fcdfc15b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdfc0b71d0 0x7fcdfc15b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 -- 172.21.15.6:0/1703235283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdfc1f4530 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde40051c0 0x7fcdfc1f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b6d50 0x7fcdfc1f3a70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00118e0 tx=0x7fcdf0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdf8ff9640 1 -- 172.21.15.6:0/1703235283 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf000faf0 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdf8ff9640 1 -- 172.21.15.6:0/1703235283 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf000fc90 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdfc1f4810 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fcdf8ff9640 1 -- 172.21.15.6:0/1703235283 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0014440 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.010+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdfc1f7590 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.012+0000 7fcdf8ff9640 1 -- 172.21.15.6:0/1703235283 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcdf0012040 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.012+0000 7fcdf8ff9640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcde806bbd0 0x7fcde806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.012+0000 7fcdf8ff9640 1 -- 172.21.15.6:0/1703235283 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcdf008b330 con 0x7fcdfc0b6d50 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.012+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcde806bbd0 0x7fcde806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.013+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcde806bbd0 0x7fcde806e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000fb0 tx=0x7fcdec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.013+0000 7fce051b6740 1 Processor -- start 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.013+0000 7fce051b6740 1 -- start start 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c0c80 0x55a159393c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce051b6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a159394160 0x55a1593c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce051b6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a1592fcc80 con 0x55a159394160 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a1592a5aa0 con 0x55a1593c1060 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x55a1592f8d20 con 0x55a1593c0c80 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a159394160 0x55a1593c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a159394160 0x55a1593c3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58084/0 (socket says 172.21.15.6:58084) 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 -- 172.21.15.6:0/922363901 learned_addr learned my addr 172.21.15.6:0/922363901 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c0c80 0x55a159393c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 -- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c0c80 msgr2=0x55a159393c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c0c80 0x55a159393c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 -- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 msgr2=0x55a1593c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fce00b70640 1 -- 172.21.15.6:0/922363901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1592c7ff0 con 0x55a159394160 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.014+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c0c80 0x55a159393c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce00b70640 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a159394160 0x55a1593c3f50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcdf402c060 tx=0x7fcdf4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376e4e25f7e0c524 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fcde1ffb640 1 -- 172.21.15.6:0/922363901 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4003970 con 0x55a159394160 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fcde1ffb640 1 -- 172.21.15.6:0/922363901 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf4003b10 con 0x55a159394160 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fcde1ffb640 1 -- 172.21.15.6:0/922363901 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4005310 con 0x55a159394160 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 -- 172.21.15.6:0/922363901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a159394160 msgr2=0x55a1593c3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a159394160 0x55a1593c3f50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcdf402c060 tx=0x7fcdf4030000 comp rx=0 tx=0).stop 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 -- 172.21.15.6:0/922363901 shutdown_connections 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a159394160 0x55a1593c3f50 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 --2- 172.21.15.6:0/922363901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c0c80 0x55a159393c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 -- 172.21.15.6:0/922363901 >> 172.21.15.6:0/922363901 conn(0x55a159395e90 msgr2=0x55a1593dea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 -- 172.21.15.6:0/922363901 shutdown_connections 2024-03-18T14:43:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.015+0000 7fce051b6740 1 -- 172.21.15.6:0/922363901 wait complete. 2024-03-18T14:43:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.016+0000 7fce051b6740 1 -- start start 2024-03-18T14:43:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.016+0000 7fce051b6740 1 Processor -- start 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce051b6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593ef720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 0x55a1593c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce051b6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a159398530 con 0x55a1593c0c80 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a1592b8ef0 con 0x55a1593c1060 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x55a15939ad70 con 0x55a1593efe20 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce051b6740 10 client.0 _ll_register_callbacks cb 0x55a1592b3dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593ef720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58098/0 (socket says 172.21.15.6:58098) 2024-03-18T14:43:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 -- 172.21.15.6:0/2233639642 learned_addr learned my addr 172.21.15.6:0/2233639642 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fce00b70640 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 0x55a1593c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 -- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 msgr2=0x55a1593c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 0x55a1593c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 -- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 msgr2=0x55a1593c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.017+0000 7fcdfbfff640 1 -- 172.21.15.6:0/2233639642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1592c5a30 con 0x55a1593c0c80 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593ef720 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcdec015770 tx=0x7fcdfc0cd620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c774d8e572c77fec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcddaffd640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcddaffd640 1 -- 172.21.15.6:0/2233639642 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec014850 con 0x55a1593c0c80 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcddaffd640 1 -- 172.21.15.6:0/2233639642 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcdec01da60 con 0x55a1593c0c80 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdf4003b10 con 0x55a1593c0c80 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcddaffd640 1 -- 172.21.15.6:0/2233639642 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fcdec01dc60 con 0x55a1593c0c80 2024-03-18T14:43:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 10 client.5721 fetch_fsmap learned FSMap version 15 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55a1590112d0 con 0x55a1593c0c80 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcddaffd640 1 -- 172.21.15.6:0/2233639642 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf4003b10 con 0x55a1593c0c80 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcddaffd640 1 -- 172.21.15.6:0/2233639642 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fcdec0149f0 con 0x55a1593c0c80 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 10 client.5721 fetch_fsmap finished waiting for FSMap version 15 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 10 client.5721 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 20 client.5721 populate_metadata read hostname 'smithi006' 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593c4fc0 0x55a1593c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 4 client.5721 mds_command: new command op to 5529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 --> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a15930a1c0 con 0x55a1593c4fc0 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.018+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593c4fc0 0x55a1593c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.019+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593c4fc0 0x55a1593c53a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c5c39e364488501 server_cookie=e86322696747af81 in_seq=0 out_seq=0 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.019+0000 7fcddaffd640 10 client.5721 ms_handle_connect on v2:172.21.15.6:6832/2677248951 2024-03-18T14:43:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.019+0000 7fcddaffd640 1 -- 172.21.15.6:0/2233639642 <== mds.0 v2:172.21.15.6:6832/2677248951 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a15930a1c0 con 0x55a1593c4fc0 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.019+0000 7fcddaffd640 10 client.5721 handle_command_reply: tid=0 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.019+0000 7fce051b6740 1 client.5721 shutdown 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 msgr2=0x55a1593ef720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593ef720 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcdec015770 tx=0x7fcdfc0cd620 comp rx=0 tx=0).stop 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 shutdown_connections 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593c4fc0 0x55a1593c53a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 0x55a1593c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 --2- 172.21.15.6:0/2233639642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593ef720 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 >> 172.21.15.6:0/2233639642 conn(0x55a159394160 msgr2=0x55a1593e1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fcddaffd640 0 client.5721 ms_handle_reset on v2:172.21.15.6:6832/2677248951 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 shutdown_connections 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 1 -- 172.21.15.6:0/2233639642 wait complete. 2024-03-18T14:43:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.020+0000 7fce051b6740 20 client.5721 trim_cache size 0 max 16384 2024-03-18T14:43:02.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.038+0000 7fce051b6740 1 Processor -- start 2024-03-18T14:43:02.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.038+0000 7fce051b6740 1 -- start start 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fce051b6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593f8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 0x55a1593b7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fce051b6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc200110 con 0x55a1593c0c80 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc1ff470 con 0x55a1593c1060 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcdfc1ffac0 con 0x55a1593efe20 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593c01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58110/0 (socket says 172.21.15.6:58110) 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 -- 172.21.15.6:0/154127952 learned_addr learned my addr 172.21.15.6:0/154127952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 -- 172.21.15.6:0/154127952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 msgr2=0x55a1593b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/154127952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 0x55a1593b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 -- 172.21.15.6:0/154127952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 msgr2=0x55a1593f8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/154127952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593f8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 -- 172.21.15.6:0/154127952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1590db6b0 con 0x55a1593c0c80 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/154127952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593c01a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fcdec014fb0 tx=0x7fcdec022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2c660ad30de02d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcde17fa640 1 -- 172.21.15.6:0/154127952 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec014850 con 0x55a1593c0c80 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcde17fa640 1 -- 172.21.15.6:0/154127952 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec027070 con 0x55a1593c0c80 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.039+0000 7fcde17fa640 1 -- 172.21.15.6:0/154127952 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec01d2a0 con 0x55a1593c0c80 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 -- 172.21.15.6:0/154127952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 msgr2=0x55a1593c01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 --2- 172.21.15.6:0/154127952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593c01a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fcdec014fb0 tx=0x7fcdec022000 comp rx=0 tx=0).stop 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 -- 172.21.15.6:0/154127952 shutdown_connections 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 --2- 172.21.15.6:0/154127952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593efe20 0x55a1593b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 --2- 172.21.15.6:0/154127952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c1060 0x55a1593f8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 --2- 172.21.15.6:0/154127952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593c0c80 0x55a1593c01a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 -- 172.21.15.6:0/154127952 >> 172.21.15.6:0/154127952 conn(0x55a159394160 msgr2=0x55a159394670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 -- 172.21.15.6:0/154127952 shutdown_connections 2024-03-18T14:43:02.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 -- 172.21.15.6:0/154127952 wait complete. 2024-03-18T14:43:02.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.040+0000 7fce051b6740 1 -- start start 2024-03-18T14:43:02.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 1 Processor -- start 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c0c80 0x55a1593f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c1060 0x55a159457440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593efe20 0x55a1593ee340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55a1593c37a0 con 0x55a1593efe20 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a1593c2a60 con 0x55a1593c0c80 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x55a1593b9340 con 0x55a1593c1060 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce051b6740 10 client.0 _ll_register_callbacks cb 0x7fcdec0077f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce00b70640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593efe20 0x55a1593ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.041+0000 7fce00b70640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593efe20 0x55a1593ee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58126/0 (socket says 172.21.15.6:58126) 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce00b70640 1 -- 172.21.15.6:0/2211420074 learned_addr learned my addr 172.21.15.6:0/2211420074 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fcdfb7fe640 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c1060 0x55a159457440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c0c80 0x55a1593f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce00b70640 1 -- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c1060 msgr2=0x55a159457440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce00b70640 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c1060 0x55a159457440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce00b70640 1 -- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c0c80 msgr2=0x55a1593f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce00b70640 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c0c80 0x55a1593f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce00b70640 1 -- 172.21.15.6:0/2211420074 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1593fd5a0 con 0x55a1593efe20 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce00b70640 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593efe20 0x55a1593ee340 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4001f40 tx=0x7fcdf4001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c22391d055b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fcddaffd640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fcddaffd640 1 -- 172.21.15.6:0/2211420074 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4005640 con 0x55a1593efe20 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fcddaffd640 1 -- 172.21.15.6:0/2211420074 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcdf4006b50 con 0x55a1593efe20 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdec027070 con 0x55a1593efe20 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fcddaffd640 1 -- 172.21.15.6:0/2211420074 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fcdf4055070 con 0x55a1593efe20 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce051b6740 10 client.5727 fetch_fsmap learned FSMap version 15 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.042+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55a159457e40 con 0x55a1593efe20 2024-03-18T14:43:02.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fcddaffd640 1 -- 172.21.15.6:0/2211420074 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec027070 con 0x55a1593efe20 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fcddaffd640 1 -- 172.21.15.6:0/2211420074 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fcdf404d020 con 0x55a1593efe20 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fce051b6740 10 client.5727 fetch_fsmap finished waiting for FSMap version 15 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fce051b6740 10 client.5727 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fce051b6740 20 client.5727 populate_metadata read hostname 'smithi006' 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fce051b6740 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593bfe30 0x55a15945af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fce051b6740 4 client.5727 mds_command: new command op to 5529 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 --> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] -- command(tid 0: {"prefix": "status"}) -- 0x55a1593df770 con 0x55a1593bfe30 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593bfe30 0x55a15945af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fcddaffd640 10 client.5727 ms_handle_connect on v2:172.21.15.6:6832/2677248951 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fcdfbfff640 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593bfe30 0x55a15945af80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad2d2e702df85b7 server_cookie=441b9573cd749a58 in_seq=0 out_seq=0 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fcddaffd640 1 -- 172.21.15.6:0/2211420074 <== mds.0 v2:172.21.15.6:6832/2677248951 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55a1593df770 con 0x55a1593bfe30 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fcddaffd640 10 client.5727 handle_command_reply: tid=0 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.043+0000 7fce051b6740 1 client.5727 shutdown 2024-03-18T14:43:02.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593efe20 msgr2=0x55a1593ee340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593efe20 0x55a1593ee340 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4001f40 tx=0x7fcdf4001510 comp rx=0 tx=0).stop 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 shutdown_connections 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:6832/2677248951,v1:172.21.15.6:6834/2677248951] conn(0x55a1593bfe30 0x55a15945af80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55a1593efe20 0x55a1593ee340 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55a1593c1060 0x55a159457440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fcddaffd640 0 client.5727 ms_handle_reset on v2:172.21.15.6:6832/2677248951 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 --2- 172.21.15.6:0/2211420074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1593c0c80 0x55a1593f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 >> 172.21.15.6:0/2211420074 conn(0x55a159394160 msgr2=0x55a1593dc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 shutdown_connections 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 1 -- 172.21.15.6:0/2211420074 wait complete. 2024-03-18T14:43:02.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce051b6740 20 client.5727 trim_cache size 0 max 16384 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a75a8f99-9e80-46f7-9da4-368af92e8063", 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5529, 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 4.7412755520000003, 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 15, 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 39, 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 39, 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 11.920372538000001 2024-03-18T14:43:02.052 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:02.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcde806bbd0 msgr2=0x7fcde806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcde806bbd0 0x7fcde806e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000fb0 tx=0x7fcdec009040 comp rx=0 tx=0).stop 2024-03-18T14:43:02.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b6d50 msgr2=0x7fcdfc1f3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b6d50 0x7fcdfc1f3a70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00118e0 tx=0x7fcdf0011da0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 shutdown_connections 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcde806bbd0 0x7fcde806e090 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdfc0b71d0 0x7fcdfc15b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdfc0b6d50 0x7fcdfc1f3a70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 --2- 172.21.15.6:0/1703235283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcde40051c0 0x7fcdfc1f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.044+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 >> 172.21.15.6:0/1703235283 conn(0x7fcdfc09ae00 msgr2=0x7fcdfc0713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.045+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 shutdown_connections 2024-03-18T14:43:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.045+0000 7fce025d3640 1 -- 172.21.15.6:0/1703235283 wait complete. 2024-03-18T14:43:02.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:43:02.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.157+0000 7fa46ba85640 1 Processor -- start 2024-03-18T14:43:02.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.158+0000 7fa46ba85640 1 -- start start 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.158+0000 7fa46ba85640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa46415bd40 0x7fa4640d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.158+0000 7fa46ba85640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4640d1d90 0x7fa4640cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.158+0000 7fa46ba85640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4640cc870 con 0x7fa4640cfcd0 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.158+0000 7fa46ba85640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa4640d0500 con 0x7fa4640d1d90 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.158+0000 7fa46ba85640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa4640d07e0 con 0x7fa46415bd40 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.158+0000 7fa469020640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4640d1d90 0x7fa4640cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469020640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4640d1d90 0x7fa4640cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58140/0 (socket says 172.21.15.6:58140) 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469020640 1 -- 172.21.15.6:0/2025584936 learned_addr learned my addr 172.21.15.6:0/2025584936 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469821640 1 --2- 172.21.15.6:0/2025584936 >> v2:172.21.15.92:3300/0 conn(0x7fa46415bd40 0x7fa4640d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469020640 1 -- 172.21.15.6:0/2025584936 >> v2:172.21.15.92:3300/0 conn(0x7fa46415bd40 msgr2=0x7fa4640d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469020640 1 --2- 172.21.15.6:0/2025584936 >> v2:172.21.15.92:3300/0 conn(0x7fa46415bd40 0x7fa4640d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469020640 1 -- 172.21.15.6:0/2025584936 >> v1:172.21.15.92:6789/0 conn(0x7fa4640cfcd0 legacy=0x7fa4640ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469020640 1 -- 172.21.15.6:0/2025584936 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4640ccb10 con 0x7fa4640d1d90 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469821640 1 --2- 172.21.15.6:0/2025584936 >> v2:172.21.15.92:3300/0 conn(0x7fa46415bd40 0x7fa4640d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.159+0000 7fa469020640 1 --2- 172.21.15.6:0/2025584936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4640d1d90 0x7fa4640cf790 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa4640cca40 tx=0x7fa45402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c060a5433f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46881f640 1 -- 172.21.15.6:0/2025584936 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454035020 con 0x7fa4640d1d90 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46881f640 1 -- 172.21.15.6:0/2025584936 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa454031870 con 0x7fa4640d1d90 2024-03-18T14:43:02.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46881f640 1 -- 172.21.15.6:0/2025584936 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454031c40 con 0x7fa4640d1d90 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 -- 172.21.15.6:0/2025584936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4640d1d90 msgr2=0x7fa4640cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 --2- 172.21.15.6:0/2025584936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4640d1d90 0x7fa4640cf790 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa4640cca40 tx=0x7fa45402b910 comp rx=0 tx=0).stop 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 -- 172.21.15.6:0/2025584936 shutdown_connections 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 --2- 172.21.15.6:0/2025584936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4640d1d90 0x7fa4640cf790 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 --2- 172.21.15.6:0/2025584936 >> v2:172.21.15.92:3300/0 conn(0x7fa46415bd40 0x7fa4640d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 -- 172.21.15.6:0/2025584936 >> 172.21.15.6:0/2025584936 conn(0x7fa46409ae20 msgr2=0x7fa46408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 -- 172.21.15.6:0/2025584936 shutdown_connections 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.160+0000 7fa46ba85640 1 -- 172.21.15.6:0/2025584936 wait complete. 2024-03-18T14:43:02.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.161+0000 7fa46ba85640 1 Processor -- start 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46ba85640 1 -- start start 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46ba85640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4640cfcd0 0x7fa464155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46ba85640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4640d1d90 0x7fa464155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46ba85640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46415bd40 0x7fa46406cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46ba85640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4640d0c40 con 0x7fa46415bd40 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46ba85640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa4640d0ac0 con 0x7fa4640d1d90 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46ba85640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa4640d0100 con 0x7fa4640cfcd0 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46a022640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46415bd40 0x7fa46406cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46a022640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46415bd40 0x7fa46406cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58148/0 (socket says 172.21.15.6:58148) 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa46a022640 1 -- 172.21.15.6:0/1955173288 learned_addr learned my addr 172.21.15.6:0/1955173288 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.162+0000 7fa469821640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4640cfcd0 0x7fa464155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa469020640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4640d1d90 0x7fa464155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46a022640 1 -- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4640cfcd0 msgr2=0x7fa464155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46a022640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4640cfcd0 0x7fa464155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46a022640 1 -- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4640d1d90 msgr2=0x7fa464155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46a022640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4640d1d90 0x7fa464155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46a022640 1 -- 172.21.15.6:0/1955173288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4640cedd0 con 0x7fa46415bd40 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46a022640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46415bd40 0x7fa46406cea0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa45c00b970 tx=0x7fa45c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa45a7fc640 1 -- 172.21.15.6:0/1955173288 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c019070 con 0x7fa46415bd40 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa45a7fc640 1 -- 172.21.15.6:0/1955173288 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45c00cb40 con 0x7fa46415bd40 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46406c400 con 0x7fa46415bd40 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa45a7fc640 1 -- 172.21.15.6:0/1955173288 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c014440 con 0x7fa46415bd40 2024-03-18T14:43:02.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.163+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46406c6a0 con 0x7fa46415bd40 2024-03-18T14:43:02.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.164+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46409fa90 con 0x7fa46415bd40 2024-03-18T14:43:02.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.164+0000 7fa45a7fc640 1 -- 172.21.15.6:0/1955173288 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa45c01c070 con 0x7fa46415bd40 2024-03-18T14:43:02.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.165+0000 7fa45a7fc640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44806bbd0 0x7fa44806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:02.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.165+0000 7fa45a7fc640 1 -- 172.21.15.6:0/1955173288 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa45c08a6d0 con 0x7fa46415bd40 2024-03-18T14:43:02.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.165+0000 7fa469821640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44806bbd0 0x7fa44806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:02.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.165+0000 7fa469821640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44806bbd0 0x7fa44806e090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa46406de10 tx=0x7fa460008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:02.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.169+0000 7fa45a7fc640 1 -- 172.21.15.6:0/1955173288 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa45c058a70 con 0x7fa46415bd40 2024-03-18T14:43:02.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.432+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa46404f620 con 0x7fa46415bd40 2024-03-18T14:43:02.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.432+0000 7fa45a7fc640 1 -- 172.21.15.6:0/1955173288 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7fa45c05cb10 con 0x7fa46415bd40 2024-03-18T14:43:02.434 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44806bbd0 msgr2=0x7fa44806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44806bbd0 0x7fa44806e090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa46406de10 tx=0x7fa460008040 comp rx=0 tx=0).stop 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46415bd40 msgr2=0x7fa46406cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46415bd40 0x7fa46406cea0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa45c00b970 tx=0x7fa45c00be30 comp rx=0 tx=0).stop 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 shutdown_connections 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44806bbd0 0x7fa44806e090 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa46415bd40 0x7fa46406cea0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4640d1d90 0x7fa464155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 --2- 172.21.15.6:0/1955173288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4640cfcd0 0x7fa464155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:02.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 >> 172.21.15.6:0/1955173288 conn(0x7fa46409ae20 msgr2=0x7fa4640b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 shutdown_connections 2024-03-18T14:43:02.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02.434+0000 7fa46ba85640 1 -- 172.21.15.6:0/1955173288 wait complete. 2024-03-18T14:43:02.446 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:43:02.447 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:02.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:02.447 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:43:02.462 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:43:02.462 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:43:02.462 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:43:02.462 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:02.462 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:43:02.462 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42583sec preferred_lft 42583sec 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:02.463 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c41d:adff:fe29:e32/64 scope link 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::d035:11ff:fe58:6e4e/64 scope link 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:02.464 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:02.465 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::9023:d7ff:fecf:2a25/64 scope link 2024-03-18T14:43:02.465 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:02.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:02.465 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:43:02.527 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:43:02.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:02.529 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:43:02.584 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:43:02.584 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:43:02.585 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:43:02.585 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:43:02.585 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:43:02.585 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:43:02.585 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:43:02.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:43:02.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:43:02.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:43:02.585 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:43:02.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:43:02.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:02.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:02.727 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:43:02.743 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T14:43:02.743 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:43:02.743 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:02.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:02.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:43:02.810 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T14:43:02.810 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:02.810 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:02.865 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:02.866 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:02.866 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:02.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:02.989 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:03.045 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:03.045 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:03.045 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:03.045 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:03.045 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:03.045 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:03.045 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:03.046 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:03.047 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:03.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:03.048 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:03.100 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:43:03.100 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T14:43:03.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:03.170 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:03.225 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:03.226 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:03.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:03.227 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:03.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:2024-03-18T14:43:03.230+0000 7f817022a480 -1 init, newargv = 0x55703ddc9fc0 newargc=15 2024-03-18T14:43:03.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[54476]: starting ceph client 2024-03-18T14:43:03.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[54476]: starting fuse 2024-03-18T14:43:03.247 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:43:03.247 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:43:03.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:43:03.248 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.0")) 2024-03-18T14:43:03.249 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:43:03.373 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.0.54476.asok 2024-03-18T14:43:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:03.385 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54476.asok 2024-03-18T14:43:03.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:03.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54476.asok status 2024-03-18T14:43:03.523 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:43:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:43:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:43:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "0", 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "54476", 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5739, 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5739 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 441628832 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 441628832 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.5739 192.168.0.1:0/441628832", 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/441628832", 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 15, 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 39, 2024-03-18T14:43:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:43:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:43:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:43:03.526 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:03.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:03.532 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:43:03.532 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:03.547 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:43:03.547 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:03.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:03.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:03.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2024-03-18T14:43:03.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.764+0000 7fa4635a9640 1 Processor -- start 2024-03-18T14:43:03.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.764+0000 7fa4635a9640 1 -- start start 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.765+0000 7fa4635a9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa45c0d1de0 0x7fa45c0d3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.765+0000 7fa4635a9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa45c0d22c0 con 0x7fa45c06c730 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.765+0000 7fa4635a9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa45c0d25a0 con 0x7fa45c15e0b0 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.765+0000 7fa4635a9640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa45c0d2880 con 0x7fa45c0d1de0 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.765+0000 7fa460b44640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa45c0d1de0 0x7fa45c0d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.765+0000 7fa461b46640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa45c06c730 0x7fa45c0b6d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37470/0 (socket says 172.21.15.6:37470) 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.765+0000 7fa461b46640 1 -- 172.21.15.6:0/3249715203 learned_addr learned my addr 172.21.15.6:0/3249715203 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.766+0000 7fa460b44640 1 -- 172.21.15.6:0/3249715203 >> v1:172.21.15.92:6789/0 conn(0x7fa45c15e0b0 legacy=0x7fa45c0b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:03.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.766+0000 7fa460b44640 1 -- 172.21.15.6:0/3249715203 >> v1:172.21.15.6:6789/0 conn(0x7fa45c06c730 legacy=0x7fa45c0b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:03.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.766+0000 7fa460b44640 1 -- 172.21.15.6:0/3249715203 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45c0d2b60 con 0x7fa45c0d1de0 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.766+0000 7fa460b44640 1 --2- 172.21.15.6:0/3249715203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0d3360 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0097a0 tx=0x7fa44c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a2afd019020c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.766+0000 7fa453fff640 1 -- 172.21.15.6:0/3249715203 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c034070 con 0x7fa45c0d1de0 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.766+0000 7fa453fff640 1 -- 172.21.15.6:0/3249715203 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa44c030610 con 0x7fa45c0d1de0 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.766+0000 7fa453fff640 1 -- 172.21.15.6:0/3249715203 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0309e0 con 0x7fa45c0d1de0 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.767+0000 7fa4635a9640 1 -- 172.21.15.6:0/3249715203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c0d1de0 msgr2=0x7fa45c0d3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.767+0000 7fa4635a9640 1 --2- 172.21.15.6:0/3249715203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0d3360 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0097a0 tx=0x7fa44c02b960 comp rx=0 tx=0).stop 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.767+0000 7fa4635a9640 1 -- 172.21.15.6:0/3249715203 shutdown_connections 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.767+0000 7fa4635a9640 1 --2- 172.21.15.6:0/3249715203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0d3360 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.767+0000 7fa4635a9640 1 -- 172.21.15.6:0/3249715203 >> 172.21.15.6:0/3249715203 conn(0x7fa45c072ce0 msgr2=0x7fa45c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:03.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.767+0000 7fa4635a9640 1 -- 172.21.15.6:0/3249715203 shutdown_connections 2024-03-18T14:43:03.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.767+0000 7fa4635a9640 1 -- 172.21.15.6:0/3249715203 wait complete. 2024-03-18T14:43:03.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.769+0000 7fa4635a9640 1 Processor -- start 2024-03-18T14:43:03.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.769+0000 7fa4635a9640 1 -- start start 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa4635a9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa45c06c730 0x7fa45c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa4635a9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa4635a9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c15e0b0 0x7fa45c0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa4635a9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c0d38a0 con 0x7fa45c15e0b0 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa4635a9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c0b7410 con 0x7fa45c0d1de0 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa4635a9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa45c14fda0 con 0x7fa45c06c730 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa461b46640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c15e0b0 0x7fa45c0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:03.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa461b46640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c15e0b0 0x7fa45c0cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53798/0 (socket says 172.21.15.6:53798) 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa461b46640 1 -- 172.21.15.6:0/2897526741 learned_addr learned my addr 172.21.15.6:0/2897526741 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa460b44640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.770+0000 7fa461345640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa45c06c730 0x7fa45c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa461b46640 1 -- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa45c06c730 msgr2=0x7fa45c0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa461b46640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa45c06c730 0x7fa45c0cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa461b46640 1 -- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa45c0d1de0 msgr2=0x7fa45c0cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa461b46640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa461b46640 1 -- 172.21.15.6:0/2897526741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45c0cb8c0 con 0x7fa45c15e0b0 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa460b44640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa461345640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa45c06c730 0x7fa45c0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa461b46640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c15e0b0 0x7fa45c0cb220 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa45400cbc0 tx=0x7fa45400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa451ffb640 1 -- 172.21.15.6:0/2897526741 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454017070 con 0x7fa45c15e0b0 2024-03-18T14:43:03.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa451ffb640 1 -- 172.21.15.6:0/2897526741 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45400ad40 con 0x7fa45c15e0b0 2024-03-18T14:43:03.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45c0cbba0 con 0x7fa45c15e0b0 2024-03-18T14:43:03.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa451ffb640 1 -- 172.21.15.6:0/2897526741 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454012600 con 0x7fa45c15e0b0 2024-03-18T14:43:03.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.771+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa45c0cc440 con 0x7fa45c15e0b0 2024-03-18T14:43:03.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.772+0000 7fa451ffb640 1 -- 172.21.15.6:0/2897526741 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa45401a070 con 0x7fa45c15e0b0 2024-03-18T14:43:03.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.772+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45c0cbd70 con 0x7fa45c15e0b0 2024-03-18T14:43:03.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.773+0000 7fa451ffb640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44006bbd0 0x7fa44006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:03.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.773+0000 7fa451ffb640 1 -- 172.21.15.6:0/2897526741 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa45408a860 con 0x7fa45c15e0b0 2024-03-18T14:43:03.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.773+0000 7fa461345640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44006bbd0 0x7fa44006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:03.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.774+0000 7fa461345640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44006bbd0 0x7fa44006e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa458006fd0 tx=0x7fa458008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:03.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.781+0000 7fa451ffb640 1 -- 172.21.15.6:0/2897526741 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa454058c00 con 0x7fa45c15e0b0 2024-03-18T14:43:03.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:03.963+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) -- 0x7fa45c08afe0 con 0x7fa45c15e0b0 2024-03-18T14:43:04.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.344+0000 7fa451ffb640 1 -- 172.21.15.6:0/2897526741 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v40) ==== 124+0+0 (secure 0 0 0) 0x7fa45405cca0 con 0x7fa45c15e0b0 2024-03-18T14:43:04.345 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'data_new' created 2024-03-18T14:43:04.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44006bbd0 msgr2=0x7fa44006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44006bbd0 0x7fa44006e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa458006fd0 tx=0x7fa458008040 comp rx=0 tx=0).stop 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c15e0b0 msgr2=0x7fa45c0cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c15e0b0 0x7fa45c0cb220 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa45400cbc0 tx=0x7fa45400a430 comp rx=0 tx=0).stop 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 shutdown_connections 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa44006bbd0 0x7fa44006e090 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa45c15e0b0 0x7fa45c0cb220 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa45c0d1de0 0x7fa45c0cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 --2- 172.21.15.6:0/2897526741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa45c06c730 0x7fa45c0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 >> 172.21.15.6:0/2897526741 conn(0x7fa45c072ce0 msgr2=0x7fa45c0c8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 shutdown_connections 2024-03-18T14:43:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.347+0000 7fa4635a9640 1 -- 172.21.15.6:0/2897526741 wait complete. 2024-03-18T14:43:04.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2024-03-18T14:43:04.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.470+0000 7f80825b3640 1 Processor -- start 2024-03-18T14:43:04.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.470+0000 7f80825b3640 1 -- start start 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f80825b3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f80825b3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f807c0b5740 0x7f807c0b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f80825b3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f807c0d1e40 0x7f807c0d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f80825b3640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f807c0b60e0 con 0x7f807c0b5740 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f80825b3640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f807c15e3b0 con 0x7f807c0d1e40 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f80825b3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f807c0b8470 con 0x7f807c15dfd0 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f807c0b5740 0x7f807c0b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f807c0b5740 0x7f807c0b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53806/0 (socket says 172.21.15.6:53806) 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 -- 172.21.15.6:0/3009878597 learned_addr learned my addr 172.21.15.6:0/3009878597 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f8080b50640 1 --2- 172.21.15.6:0/3009878597 >> v2:172.21.15.92:3300/0 conn(0x7f807c0d1e40 0x7f807c0d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807bfff640 1 --2- 172.21.15.6:0/3009878597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 -- 172.21.15.6:0/3009878597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 msgr2=0x7f807c0b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:04.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 --2- 172.21.15.6:0/3009878597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 -- 172.21.15.6:0/3009878597 >> v2:172.21.15.92:3300/0 conn(0x7f807c0d1e40 msgr2=0x7f807c0d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:04.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 --2- 172.21.15.6:0/3009878597 >> v2:172.21.15.92:3300/0 conn(0x7f807c0d1e40 0x7f807c0d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807b7fe640 1 -- 172.21.15.6:0/3009878597 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c0b8750 con 0x7f807c0b5740 2024-03-18T14:43:04.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f807bfff640 1 --2- 172.21.15.6:0/3009878597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:04.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.471+0000 7f8080b50640 1 --2- 172.21.15.6:0/3009878597 >> v2:172.21.15.92:3300/0 conn(0x7f807c0d1e40 0x7f807c0d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.472+0000 7f807b7fe640 1 --2- 172.21.15.6:0/3009878597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0b5740 0x7f807c0b5ba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f807c0b8ff0 tx=0x7f807002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfec948f8a6b73af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.472+0000 7f807affd640 1 -- 172.21.15.6:0/3009878597 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070035020 con 0x7f807c0b5740 2024-03-18T14:43:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.472+0000 7f807affd640 1 -- 172.21.15.6:0/3009878597 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070031870 con 0x7f807c0b5740 2024-03-18T14:43:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.472+0000 7f807affd640 1 -- 172.21.15.6:0/3009878597 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070031c40 con 0x7f807c0b5740 2024-03-18T14:43:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 -- 172.21.15.6:0/3009878597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0b5740 msgr2=0x7f807c0b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:04.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 --2- 172.21.15.6:0/3009878597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0b5740 0x7f807c0b5ba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f807c0b8ff0 tx=0x7f807002b910 comp rx=0 tx=0).stop 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 -- 172.21.15.6:0/3009878597 shutdown_connections 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 --2- 172.21.15.6:0/3009878597 >> v2:172.21.15.92:3300/0 conn(0x7f807c0d1e40 0x7f807c0d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 --2- 172.21.15.6:0/3009878597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0b5740 0x7f807c0b5ba0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 --2- 172.21.15.6:0/3009878597 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 -- 172.21.15.6:0/3009878597 >> 172.21.15.6:0/3009878597 conn(0x7f807c072d10 msgr2=0x7f807c073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 -- 172.21.15.6:0/3009878597 shutdown_connections 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.473+0000 7f80825b3640 1 -- 172.21.15.6:0/3009878597 wait complete. 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.474+0000 7f80825b3640 1 Processor -- start 2024-03-18T14:43:04.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.474+0000 7f80825b3640 1 -- start start 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f80825b3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f807c0b5740 0x7f807c152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f80825b3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0d1e40 0x7f807c0ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f80825b3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f80825b3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f807c0b9b90 con 0x7f807c0d1e40 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f80825b3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f807c0b8d10 con 0x7f807c0b5740 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f80825b3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f807c0d2810 con 0x7f807c15dfd0 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0d1e40 0x7f807c0ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:04.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0d1e40 0x7f807c0ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53810/0 (socket says 172.21.15.6:53810) 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 -- 172.21.15.6:0/4266740137 learned_addr learned my addr 172.21.15.6:0/4266740137 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f8080b50640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807bfff640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f807c0b5740 0x7f807c152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 -- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 msgr2=0x7f807c0cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 -- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f807c0b5740 msgr2=0x7f807c152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f807c0b5740 0x7f807c152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 -- 172.21.15.6:0/4266740137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c160390 con 0x7f807c0d1e40 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.475+0000 7f807b7fe640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0d1e40 0x7f807c0ccda0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f807002b950 tx=0x7f80700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.476+0000 7f8078ff9640 1 -- 172.21.15.6:0/4266740137 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070040070 con 0x7f807c0d1e40 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.476+0000 7f8078ff9640 1 -- 172.21.15.6:0/4266740137 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070030d00 con 0x7f807c0d1e40 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.476+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807c160670 con 0x7f807c0d1e40 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.476+0000 7f8078ff9640 1 -- 172.21.15.6:0/4266740137 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070035040 con 0x7f807c0d1e40 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.476+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807c160c30 con 0x7f807c0d1e40 2024-03-18T14:43:04.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.477+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807c0751b0 con 0x7f807c0d1e40 2024-03-18T14:43:04.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.478+0000 7f8078ff9640 1 -- 172.21.15.6:0/4266740137 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f807004b060 con 0x7f807c0d1e40 2024-03-18T14:43:04.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.478+0000 7f8078ff9640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f806806bbd0 0x7f806806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:04.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.478+0000 7f8078ff9640 1 -- 172.21.15.6:0/4266740137 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6464+0+0 (secure 0 0 0) 0x7f80700ac9e0 con 0x7f807c0d1e40 2024-03-18T14:43:04.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.479+0000 7f807bfff640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f806806bbd0 0x7f806806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:04.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.479+0000 7f807bfff640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f806806bbd0 0x7f806806e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f806c006fd0 tx=0x7f806c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.483+0000 7f8078ff9640 1 -- 172.21.15.6:0/4266740137 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f807007ab90 con 0x7f807c0d1e40 2024-03-18T14:43:04.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:04.714+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) -- 0x7f807c06b910 con 0x7f807c0d1e40 2024-03-18T14:43:05.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.355+0000 7f8078ff9640 1 -- 172.21.15.6:0/4266740137 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 7 to fsmap v16) ==== 131+0+0 (secure 0 0 0) 0x7f807007ec30 con 0x7f807c0d1e40 2024-03-18T14:43:05.357 INFO:teuthology.orchestra.run.smithi006.stderr:added data pool 7 to fsmap 2024-03-18T14:43:05.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.358+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f806806bbd0 msgr2=0x7f806806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.358+0000 7f80825b3640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f806806bbd0 0x7f806806e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f806c006fd0 tx=0x7f806c008040 comp rx=0 tx=0).stop 2024-03-18T14:43:05.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.358+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0d1e40 msgr2=0x7f807c0ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.358+0000 7f80825b3640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0d1e40 0x7f807c0ccda0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f807002b950 tx=0x7f80700046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.359+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 shutdown_connections 2024-03-18T14:43:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.359+0000 7f80825b3640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f806806bbd0 0x7f806806e090 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.359+0000 7f80825b3640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f807c15dfd0 0x7f807c0cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.359+0000 7f80825b3640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f807c0d1e40 0x7f807c0ccda0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.359+0000 7f80825b3640 1 --2- 172.21.15.6:0/4266740137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f807c0b5740 0x7f807c152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.359+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 >> 172.21.15.6:0/4266740137 conn(0x7f807c072d10 msgr2=0x7f807c0b7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:05.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.360+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 shutdown_connections 2024-03-18T14:43:05.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.361+0000 7f80825b3640 1 -- 172.21.15.6:0/4266740137 wait complete. 2024-03-18T14:43:05.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:05.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:43:05.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:43:05.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.475+0000 7fdd4771b640 1 Processor -- start 2024-03-18T14:43:05.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.475+0000 7fdd4771b640 1 -- start start 2024-03-18T14:43:05.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd4771b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd400d1f60 0x7fdd400cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd4771b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdd400cfcd0 0x7fdd400cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd4771b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd400cc870 con 0x7fdd4015bd40 2024-03-18T14:43:05.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd4771b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fdd400d0500 con 0x7fdd400d1f60 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd4771b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdd400d07e0 con 0x7fdd400cfcd0 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd44cb6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd400d1f60 0x7fdd400cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd44cb6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdd400d1f60 0x7fdd400cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53818/0 (socket says 172.21.15.6:53818) 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.476+0000 7fdd44cb6640 1 -- 172.21.15.6:0/2548127439 learned_addr learned my addr 172.21.15.6:0/2548127439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.477+0000 7fdd45cb8640 1 --2- 172.21.15.6:0/2548127439 >> v2:172.21.15.92:3300/0 conn(0x7fdd400cfcd0 0x7fdd400cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.477+0000 7fdd44cb6640 1 -- 172.21.15.6:0/2548127439 >> v2:172.21.15.92:3300/0 conn(0x7fdd400cfcd0 msgr2=0x7fdd400cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.477+0000 7fdd44cb6640 1 --2- 172.21.15.6:0/2548127439 >> v2:172.21.15.92:3300/0 conn(0x7fdd400cfcd0 0x7fdd400cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.477+0000 7fdd44cb6640 1 -- 172.21.15.6:0/2548127439 >> v1:172.21.15.92:6789/0 conn(0x7fdd4015bd40 legacy=0x7fdd400d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.477+0000 7fdd44cb6640 1 -- 172.21.15.6:0/2548127439 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd400ccb10 con 0x7fdd400d1f60 2024-03-18T14:43:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.477+0000 7fdd44cb6640 1 --2- 172.21.15.6:0/2548127439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400d1f60 0x7fdd400cf790 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdd400cca40 tx=0x7fdd3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f844595cbd947ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:05.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd37fff640 1 -- 172.21.15.6:0/2548127439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30035020 con 0x7fdd400d1f60 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd37fff640 1 -- 172.21.15.6:0/2548127439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd30031870 con 0x7fdd400d1f60 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd37fff640 1 -- 172.21.15.6:0/2548127439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30031c40 con 0x7fdd400d1f60 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 -- 172.21.15.6:0/2548127439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400d1f60 msgr2=0x7fdd400cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 --2- 172.21.15.6:0/2548127439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400d1f60 0x7fdd400cf790 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdd400cca40 tx=0x7fdd3002b910 comp rx=0 tx=0).stop 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 -- 172.21.15.6:0/2548127439 shutdown_connections 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 --2- 172.21.15.6:0/2548127439 >> v2:172.21.15.92:3300/0 conn(0x7fdd400cfcd0 0x7fdd400cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 --2- 172.21.15.6:0/2548127439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400d1f60 0x7fdd400cf790 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 -- 172.21.15.6:0/2548127439 >> 172.21.15.6:0/2548127439 conn(0x7fdd4009ae20 msgr2=0x7fdd4008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 -- 172.21.15.6:0/2548127439 shutdown_connections 2024-03-18T14:43:05.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.478+0000 7fdd4771b640 1 -- 172.21.15.6:0/2548127439 wait complete. 2024-03-18T14:43:05.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 Processor -- start 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 -- start start 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400cfcd0 0x7fdd40154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd400d1f60 0x7fdd400cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4015bd40 0x7fdd400ce480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdd400d0280 con 0x7fdd400cfcd0 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdd400d0100 con 0x7fdd400d1f60 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd4771b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4015c120 con 0x7fdd4015bd40 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.480+0000 7fdd454b7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400cfcd0 0x7fdd40154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400cfcd0 0x7fdd40154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53826/0 (socket says 172.21.15.6:53826) 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 -- 172.21.15.6:0/1014333871 learned_addr learned my addr 172.21.15.6:0/1014333871 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd44cb6640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd400d1f60 0x7fdd400cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd45cb8640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4015bd40 0x7fdd400ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 -- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4015bd40 msgr2=0x7fdd400ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4015bd40 0x7fdd400ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 -- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd400d1f60 msgr2=0x7fdd400cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd400d1f60 0x7fdd400cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 -- 172.21.15.6:0/1014333871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd401f28a0 con 0x7fdd400cfcd0 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd454b7640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400cfcd0 0x7fdd40154750 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00c3c0 tx=0x7fdd3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd35ffb640 1 -- 172.21.15.6:0/1014333871 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0052d0 con 0x7fdd400cfcd0 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd35ffb640 1 -- 172.21.15.6:0/1014333871 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd3c005490 con 0x7fdd400cfcd0 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd401f1d40 con 0x7fdd400cfcd0 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.481+0000 7fdd35ffb640 1 -- 172.21.15.6:0/1014333871 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c007260 con 0x7fdd400cfcd0 2024-03-18T14:43:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.482+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd401f2300 con 0x7fdd400cfcd0 2024-03-18T14:43:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.483+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd4009fa90 con 0x7fdd400cfcd0 2024-03-18T14:43:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.483+0000 7fdd35ffb640 1 -- 172.21.15.6:0/1014333871 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdd3c007400 con 0x7fdd400cfcd0 2024-03-18T14:43:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.484+0000 7fdd35ffb640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd2406bbd0 0x7fdd2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.484+0000 7fdd35ffb640 1 -- 172.21.15.6:0/1014333871 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6496+0+0 (secure 0 0 0) 0x7fdd3c08a0c0 con 0x7fdd400cfcd0 2024-03-18T14:43:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.484+0000 7fdd44cb6640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd2406bbd0 0x7fdd2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.484+0000 7fdd44cb6640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd2406bbd0 0x7fdd2406e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd30036000 tx=0x7fdd3003c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:05.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.491+0000 7fdd35ffb640 1 -- 172.21.15.6:0/1014333871 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdd3c0582e0 con 0x7fdd400cfcd0 2024-03-18T14:43:05.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.756+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd4004f620 con 0x7fdd400cfcd0 2024-03-18T14:43:05.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.757+0000 7fdd35ffb640 1 -- 172.21.15.6:0/1014333871 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5171 (secure 0 0 0) 0x7fdd3c05c380 con 0x7fdd400cfcd0 2024-03-18T14:43:05.759 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:05.759 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:42:56.672813+0000","modified":"2024-03-18T14:43:04.715542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5529":{"gid":5529,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2677248951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2677248951},{"type":"v1","addr":"172.21.15.6:6834","nonce":2677248951}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5529,"qdb_cluster":[5529]},"id":2}]} 2024-03-18T14:43:05.759 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd2406bbd0 msgr2=0x7fdd2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd2406bbd0 0x7fdd2406e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd30036000 tx=0x7fdd3003c040 comp rx=0 tx=0).stop 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400cfcd0 msgr2=0x7fdd40154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400cfcd0 0x7fdd40154750 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00c3c0 tx=0x7fdd3c00c880 comp rx=0 tx=0).stop 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 shutdown_connections 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdd2406bbd0 0x7fdd2406e090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4015bd40 0x7fdd400ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd400d1f60 0x7fdd400cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 --2- 172.21.15.6:0/1014333871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd400cfcd0 0x7fdd40154750 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 >> 172.21.15.6:0/1014333871 conn(0x7fdd4009ae20 msgr2=0x7fdd4015b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:05.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 shutdown_connections 2024-03-18T14:43:05.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.759+0000 7fdd4771b640 1 -- 172.21.15.6:0/1014333871 wait complete. 2024-03-18T14:43:05.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:05.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.871+0000 7f2cc95e6640 1 Processor -- start 2024-03-18T14:43:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.871+0000 7f2cc95e6640 1 -- start start 2024-03-18T14:43:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc95e6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cc415e020 0x7f2cc4162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc95e6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2cc40ced00 0x7f2cc40d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc95e6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc40d3210 0x7f2cc40d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc95e6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f2cc40b9660 con 0x7f2cc415e020 2024-03-18T14:43:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc95e6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2cc40ce390 con 0x7f2cc40ced00 2024-03-18T14:43:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc95e6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc40b9900 con 0x7f2cc40d3210 2024-03-18T14:43:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc2ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cc415e020 0x7f2cc4162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc2ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cc415e020 0x7f2cc4162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53838/0 (socket says 172.21.15.6:53838) 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/3193070043 learned_addr learned my addr 172.21.15.6:0/3193070043 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc37fe640 1 --2- 172.21.15.6:0/3193070043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc40d3210 0x7f2cc40d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc27fc640 1 --2- 172.21.15.6:0/3193070043 >> v2:172.21.15.92:3300/0 conn(0x7f2cc40ced00 0x7f2cc40d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/3193070043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc40d3210 msgr2=0x7f2cc40d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.872+0000 7f2cc2ffd640 1 --2- 172.21.15.6:0/3193070043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc40d3210 0x7f2cc40d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.873+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/3193070043 >> v2:172.21.15.92:3300/0 conn(0x7f2cc40ced00 msgr2=0x7f2cc40d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.873+0000 7f2cc2ffd640 1 --2- 172.21.15.6:0/3193070043 >> v2:172.21.15.92:3300/0 conn(0x7f2cc40ced00 0x7f2cc40d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.873+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/3193070043 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc40b9be0 con 0x7f2cc415e020 2024-03-18T14:43:05.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.873+0000 7f2cc2ffd640 1 --2- 172.21.15.6:0/3193070043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc415e020 0x7f2cc4162380 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40d4da0 tx=0x7f2cac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9ec37223e88605 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:05.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.873+0000 7f2cc1ffb640 1 -- 172.21.15.6:0/3193070043 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac02e070 con 0x7f2cc415e020 2024-03-18T14:43:05.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.873+0000 7f2cc1ffb640 1 -- 172.21.15.6:0/3193070043 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cac037070 con 0x7f2cc415e020 2024-03-18T14:43:05.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.873+0000 7f2cc1ffb640 1 -- 172.21.15.6:0/3193070043 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac031a00 con 0x7f2cc415e020 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 -- 172.21.15.6:0/3193070043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc415e020 msgr2=0x7f2cc4162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/3193070043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc415e020 0x7f2cc4162380 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40d4da0 tx=0x7f2cac02b880 comp rx=0 tx=0).stop 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 -- 172.21.15.6:0/3193070043 shutdown_connections 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/3193070043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc40d3210 0x7f2cc40d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/3193070043 >> v2:172.21.15.92:3300/0 conn(0x7f2cc40ced00 0x7f2cc40d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/3193070043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc415e020 0x7f2cc4162380 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 -- 172.21.15.6:0/3193070043 >> 172.21.15.6:0/3193070043 conn(0x7f2cc4072ce0 msgr2=0x7f2cc40730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 -- 172.21.15.6:0/3193070043 shutdown_connections 2024-03-18T14:43:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.874+0000 7f2cc95e6640 1 -- 172.21.15.6:0/3193070043 wait complete. 2024-03-18T14:43:05.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.875+0000 7f2cc95e6640 1 Processor -- start 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.875+0000 7f2cc95e6640 1 -- start start 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc95e6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc40ced00 0x7f2cc41f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc95e6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cc40d3210 0x7f2cc4160040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc95e6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc415e020 0x7f2cc40b5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc95e6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc40cf440 con 0x7f2cc40ced00 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc95e6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc40cf2c0 con 0x7f2cc40d3210 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc95e6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc40d20a0 con 0x7f2cc415e020 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc2ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc40ced00 0x7f2cc41f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc2ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc40ced00 0x7f2cc41f50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53852/0 (socket says 172.21.15.6:53852) 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.876+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/1165307199 learned_addr learned my addr 172.21.15.6:0/1165307199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc27fc640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cc40d3210 0x7f2cc4160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc37fe640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc415e020 0x7f2cc40b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc415e020 msgr2=0x7f2cc40b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc2ffd640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc415e020 0x7f2cc40b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cc40d3210 msgr2=0x7f2cc4160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc2ffd640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cc40d3210 0x7f2cc4160040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc2ffd640 1 -- 172.21.15.6:0/1165307199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc41f55f0 con 0x7f2cc40ced00 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc37fe640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc415e020 0x7f2cc40b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc27fc640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cc40d3210 0x7f2cc4160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc2ffd640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc40ced00 0x7f2cc41f50b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2cac02b8c0 tx=0x7f2cac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cbb7fe640 1 -- 172.21.15.6:0/1165307199 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac032670 con 0x7f2cc40ced00 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cbb7fe640 1 -- 172.21.15.6:0/1165307199 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cac032830 con 0x7f2cc40ced00 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc4160580 con 0x7f2cc40ced00 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cbb7fe640 1 -- 172.21.15.6:0/1165307199 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac032c20 con 0x7f2cc40ced00 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.877+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc40b3c50 con 0x7f2cc40ced00 2024-03-18T14:43:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.878+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc40b69e0 con 0x7f2cc40ced00 2024-03-18T14:43:05.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.878+0000 7f2cbb7fe640 1 -- 172.21.15.6:0/1165307199 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2cac04c060 con 0x7f2cc40ced00 2024-03-18T14:43:05.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.879+0000 7f2cbb7fe640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ca406bbd0 0x7f2ca406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:05.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.879+0000 7f2cbb7fe640 1 -- 172.21.15.6:0/1165307199 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6496+0+0 (secure 0 0 0) 0x7f2cac0ac640 con 0x7f2cc40ced00 2024-03-18T14:43:05.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.879+0000 7f2cc27fc640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ca406bbd0 0x7f2ca406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:05.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.880+0000 7f2cc27fc640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ca406bbd0 0x7f2ca406e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4161470 tx=0x7f2cb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:05.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:05.883+0000 7f2cbb7fe640 1 -- 172.21.15.6:0/1165307199 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2cc40b69e0 con 0x7f2cc40ced00 2024-03-18T14:43:06.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.074+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2cc40b52e0 con 0x7f2cc40ced00 2024-03-18T14:43:06.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.075+0000 7f2cbb7fe640 1 -- 172.21.15.6:0/1165307199 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+18671 (secure 0 0 0) 0x7f2cc41f55f0 con 0x7f2cc40ced00 2024-03-18T14:43:06.077 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:06.077 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:04.715301+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:06.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ca406bbd0 msgr2=0x7f2ca406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ca406bbd0 0x7f2ca406e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4161470 tx=0x7f2cb4008040 comp rx=0 tx=0).stop 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc40ced00 msgr2=0x7f2cc41f50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc40ced00 0x7f2cc41f50b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2cac02b8c0 tx=0x7f2cac0046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 shutdown_connections 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2ca406bbd0 0x7f2ca406e090 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2cc415e020 0x7f2cc40b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cc40d3210 0x7f2cc4160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 --2- 172.21.15.6:0/1165307199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc40ced00 0x7f2cc41f50b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 >> 172.21.15.6:0/1165307199 conn(0x7f2cc4072ce0 msgr2=0x7f2cc41620a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 shutdown_connections 2024-03-18T14:43:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.077+0000 7f2cc95e6640 1 -- 172.21.15.6:0/1165307199 wait complete. 2024-03-18T14:43:06.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:06.089 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2024-03-18T14:43:06.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:06.108 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2024-03-18T14:43:06.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-18T14:43:06.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.311+0000 7fbe69f37640 1 Processor -- start 2024-03-18T14:43:06.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.311+0000 7fbe69f37640 1 -- start start 2024-03-18T14:43:06.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe69f37640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbe640cf850 0x7fbe640cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe69f37640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 0x7fbe640b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe69f37640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe640ce590 con 0x7fbe6415ac60 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe69f37640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbe640cdd40 con 0x7fbe640cf850 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe69f37640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbe640cf3c0 con 0x7fbe640cd8f0 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe637fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbe6415ac60 0x7fbe640cc170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37480/0 (socket says 172.21.15.6:37480) 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe637fe640 1 -- 172.21.15.6:0/1568818765 learned_addr learned my addr 172.21.15.6:0/1568818765 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe63fff640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 0x7fbe640b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe62ffd640 1 --2- 172.21.15.6:0/1568818765 >> v2:172.21.15.92:3300/0 conn(0x7fbe640cf850 0x7fbe640cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234434618 0 0) 0x7fbe640ce590 con 0x7fbe6415ac60 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe4c003520 con 0x7fbe6415ac60 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 634264259 0 0) 0x7fbe4c003520 con 0x7fbe6415ac60 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 msgr2=0x7fbe640b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 0x7fbe640b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 >> v2:172.21.15.92:3300/0 conn(0x7fbe640cf850 msgr2=0x7fbe640cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 --2- 172.21.15.6:0/1568818765 >> v2:172.21.15.92:3300/0 conn(0x7fbe640cf850 0x7fbe640cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe640ce830 con 0x7fbe6415ac60 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fbe58001670 con 0x7fbe6415ac60 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 >> v1:172.21.15.6:6789/0 conn(0x7fbe6415ac60 legacy=0x7fbe640cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.312+0000 7fbe627fc640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 0x7fbe4c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe627fc640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c003f30 0x7fbe4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe627fc640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe4c004d40 0x7fbe4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640cdd40 con 0x7fbe4c004d40 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640cf3c0 con 0x7fbe4c003f30 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbe4c00b9c0 con 0x7fbe640cd8f0 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe63fff640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe4c004d40 0x7fbe4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe637fe640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 0x7fbe4c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe62ffd640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c003f30 0x7fbe4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe63fff640 1 -- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 msgr2=0x7fbe4c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe63fff640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe640cd8f0 0x7fbe4c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe63fff640 1 -- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c003f30 msgr2=0x7fbe4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe63fff640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c003f30 0x7fbe4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.313+0000 7fbe63fff640 1 -- 172.21.15.6:0/1568818765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4c003520 con 0x7fbe4c004d40 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe63fff640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe4c004d40 0x7fbe4c0051c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c010350 tx=0x7fbe4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bcdbe25f1c71937 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00be20 con 0x7fbe4c004d40 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5c02ee00 con 0x7fbe4c004d40 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe627fc640 1 -- 172.21.15.6:0/1568818765 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00b310 con 0x7fbe4c004d40 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe69f37640 1 -- 172.21.15.6:0/1568818765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe4c004d40 msgr2=0x7fbe4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe69f37640 1 --2- 172.21.15.6:0/1568818765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe4c004d40 0x7fbe4c0051c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c010350 tx=0x7fbe4c005830 comp rx=0 tx=0).stop 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe63fff640 1 -- 172.21.15.6:0/1568818765 reap_dead start 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe69f37640 1 -- 172.21.15.6:0/1568818765 shutdown_connections 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe69f37640 1 -- 172.21.15.6:0/1568818765 >> 172.21.15.6:0/1568818765 conn(0x7fbe64099a50 msgr2=0x7fbe640d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:06.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe69f37640 1 -- 172.21.15.6:0/1568818765 shutdown_connections 2024-03-18T14:43:06.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.314+0000 7fbe69f37640 1 -- 172.21.15.6:0/1568818765 wait complete. 2024-03-18T14:43:06.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 Processor -- start 2024-03-18T14:43:06.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 -- start start 2024-03-18T14:43:06.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004d40 0x7fbe6414acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe640cc540 0x7fbe640cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe6415c6b0 0x7fbe6415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640cf050 con 0x7fbe640cc540 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640cead0 con 0x7fbe4c004d40 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.316+0000 7fbe69f37640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbe6415b040 con 0x7fbe6415c6b0 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe640cc540 0x7fbe640cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe640cc540 0x7fbe640cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53880/0 (socket says 172.21.15.6:53880) 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 -- 172.21.15.6:0/3993905524 learned_addr learned my addr 172.21.15.6:0/3993905524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe637fe640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004d40 0x7fbe6414acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe63fff640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe6415c6b0 0x7fbe6415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 -- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe6415c6b0 msgr2=0x7fbe6415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe6415c6b0 0x7fbe6415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 -- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004d40 msgr2=0x7fbe6414acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004d40 0x7fbe6414acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 -- 172.21.15.6:0/3993905524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6415d070 con 0x7fbe640cc540 2024-03-18T14:43:06.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe63fff640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe6415c6b0 0x7fbe6415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe637fe640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004d40 0x7fbe6414acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe62ffd640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe640cc540 0x7fbe640cc9a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbe54011960 tx=0x7fbe54011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe4bfff640 1 -- 172.21.15.6:0/3993905524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5400faf0 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe4bfff640 1 -- 172.21.15.6:0/3993905524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5400fc90 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6415d2e0 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe4bfff640 1 -- 172.21.15.6:0/3993905524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54014440 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.317+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6415d8a0 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.318+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe640ccee0 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.319+0000 7fbe4bfff640 1 -- 172.21.15.6:0/3993905524 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbe54012040 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.319+0000 7fbe4bfff640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe5006bbd0 0x7fbe5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.319+0000 7fbe4bfff640 1 -- 172.21.15.6:0/3993905524 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6496+0+0 (secure 0 0 0) 0x7fbe5408b460 con 0x7fbe640cc540 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.319+0000 7fbe637fe640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe5006bbd0 0x7fbe5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.320+0000 7fbe637fe640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe5006bbd0 0x7fbe5006e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbe58007fd0 tx=0x7fbe58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:06.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.324+0000 7fbe4bfff640 1 -- 172.21.15.6:0/3993905524 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbe54019030 con 0x7fbe640cc540 2024-03-18T14:43:06.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.580+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7fbe640751b0 con 0x7fbe640cc540 2024-03-18T14:43:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.582+0000 7fbe4bfff640 1 -- 172.21.15.6:0/3993905524 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v12) ==== 210+0+167 (secure 0 0 0) 0x7fbe540595f0 con 0x7fbe640cc540 2024-03-18T14:43:06.583 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:43:06.583 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:43:06.583 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow rw" 2024-03-18T14:43:06.583 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:43:06.583 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-18T14:43:06.584 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe5006bbd0 msgr2=0x7fbe5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe5006bbd0 0x7fbe5006e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbe58007fd0 tx=0x7fbe58009040 comp rx=0 tx=0).stop 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe640cc540 msgr2=0x7fbe640cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe640cc540 0x7fbe640cc9a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbe54011960 tx=0x7fbe54011e20 comp rx=0 tx=0).stop 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 shutdown_connections 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbe5006bbd0 0x7fbe5006e090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe6415c6b0 0x7fbe6415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe640cc540 0x7fbe640cc9a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 --2- 172.21.15.6:0/3993905524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004d40 0x7fbe6414acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 >> 172.21.15.6:0/3993905524 conn(0x7fbe64099a50 msgr2=0x7fbe640cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:06.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 shutdown_connections 2024-03-18T14:43:06.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.584+0000 7fbe69f37640 1 -- 172.21.15.6:0/3993905524 wait complete. 2024-03-18T14:43:06.599 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:06.599 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:06.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:06.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.753+0000 7f4b67990640 1 Processor -- start 2024-03-18T14:43:06.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.753+0000 7f4b67990640 1 -- start start 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b67990640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4b6015bd40 0x7f4b600d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b67990640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4b600d1d90 0x7f4b600cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b67990640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b600cfcd0 0x7f4b600ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b67990640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4b600cc870 con 0x7f4b6015bd40 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b67990640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4b600d0500 con 0x7f4b600d1d90 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b67990640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4b600d07e0 con 0x7f4b600cfcd0 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b6572c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4b6015bd40 0x7f4b600d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b6572c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4b6015bd40 0x7f4b600d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53888/0 (socket says 172.21.15.6:53888) 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b6572c640 1 -- 172.21.15.6:0/3260330070 learned_addr learned my addr 172.21.15.6:0/3260330070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b6572c640 1 -- 172.21.15.6:0/3260330070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b600cfcd0 msgr2=0x7f4b600ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.754+0000 7f4b64f2b640 1 --2- 172.21.15.6:0/3260330070 >> v2:172.21.15.92:3300/0 conn(0x7f4b600d1d90 0x7f4b600cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b65f2d640 1 --2- 172.21.15.6:0/3260330070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b600cfcd0 0x7f4b600ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b6572c640 1 --2- 172.21.15.6:0/3260330070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b600cfcd0 0x7f4b600ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b6572c640 1 -- 172.21.15.6:0/3260330070 >> v2:172.21.15.92:3300/0 conn(0x7f4b600d1d90 msgr2=0x7f4b600cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b6572c640 1 --2- 172.21.15.6:0/3260330070 >> v2:172.21.15.92:3300/0 conn(0x7f4b600d1d90 0x7f4b600cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b6572c640 1 -- 172.21.15.6:0/3260330070 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b600ccb10 con 0x7f4b6015bd40 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b64f2b640 1 --2- 172.21.15.6:0/3260330070 >> v2:172.21.15.92:3300/0 conn(0x7f4b600d1d90 0x7f4b600cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b65f2d640 1 --2- 172.21.15.6:0/3260330070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b600cfcd0 0x7f4b600ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b6572c640 1 --2- 172.21.15.6:0/3260330070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b6015bd40 0x7f4b600d1850 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4b60067f50 tx=0x7f4b5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d272195d37fc681 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.755+0000 7f4b53fff640 1 -- 172.21.15.6:0/3260330070 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c035020 con 0x7f4b6015bd40 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b53fff640 1 -- 172.21.15.6:0/3260330070 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b5c031870 con 0x7f4b6015bd40 2024-03-18T14:43:06.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b53fff640 1 -- 172.21.15.6:0/3260330070 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c031c40 con 0x7f4b6015bd40 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 -- 172.21.15.6:0/3260330070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b6015bd40 msgr2=0x7f4b600d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 --2- 172.21.15.6:0/3260330070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b6015bd40 0x7f4b600d1850 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4b60067f50 tx=0x7f4b5c02b910 comp rx=0 tx=0).stop 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 -- 172.21.15.6:0/3260330070 shutdown_connections 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 --2- 172.21.15.6:0/3260330070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b600cfcd0 0x7f4b600ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 --2- 172.21.15.6:0/3260330070 >> v2:172.21.15.92:3300/0 conn(0x7f4b600d1d90 0x7f4b600cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 --2- 172.21.15.6:0/3260330070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b6015bd40 0x7f4b600d1850 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 -- 172.21.15.6:0/3260330070 >> 172.21.15.6:0/3260330070 conn(0x7f4b6009ae20 msgr2=0x7f4b6008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 -- 172.21.15.6:0/3260330070 shutdown_connections 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.756+0000 7f4b67990640 1 -- 172.21.15.6:0/3260330070 wait complete. 2024-03-18T14:43:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.757+0000 7f4b67990640 1 Processor -- start 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.757+0000 7f4b67990640 1 -- start start 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b67990640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b600cfcd0 0x7f4b600b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b67990640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b600d1d90 0x7f4b601f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b67990640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b6015bd40 0x7f4b601f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b67990640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4b600d0c40 con 0x7f4b600d1d90 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b67990640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4b600d0ac0 con 0x7f4b600cfcd0 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b67990640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4b600d0140 con 0x7f4b6015bd40 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b600d1d90 0x7f4b601f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b600d1d90 0x7f4b601f4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53896/0 (socket says 172.21.15.6:53896) 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 -- 172.21.15.6:0/1538357720 learned_addr learned my addr 172.21.15.6:0/1538357720 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b6572c640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b600cfcd0 0x7f4b600b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b65f2d640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b6015bd40 0x7f4b601f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 -- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b6015bd40 msgr2=0x7f4b601f54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b6015bd40 0x7f4b601f54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 -- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b600cfcd0 msgr2=0x7f4b600b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b600cfcd0 0x7f4b600b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.758+0000 7f4b64f2b640 1 -- 172.21.15.6:0/1538357720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b600aefb0 con 0x7f4b600d1d90 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.759+0000 7f4b64f2b640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b600d1d90 0x7f4b601f4f80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400c820 tx=0x7f4b5400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.759+0000 7f4b51ffb640 1 -- 172.21.15.6:0/1538357720 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b54015070 con 0x7f4b600d1d90 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.759+0000 7f4b51ffb640 1 -- 172.21.15.6:0/1538357720 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b54007a30 con 0x7f4b600d1d90 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.759+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b600b1480 con 0x7f4b600d1d90 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.759+0000 7f4b51ffb640 1 -- 172.21.15.6:0/1538357720 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b54007e00 con 0x7f4b600d1d90 2024-03-18T14:43:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.759+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b600ae710 con 0x7f4b600d1d90 2024-03-18T14:43:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.760+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b600c50f0 con 0x7f4b600d1d90 2024-03-18T14:43:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.761+0000 7f4b51ffb640 1 -- 172.21.15.6:0/1538357720 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4b54018070 con 0x7f4b600d1d90 2024-03-18T14:43:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.761+0000 7f4b51ffb640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4b4406bbd0 0x7f4b4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.761+0000 7f4b51ffb640 1 -- 172.21.15.6:0/1538357720 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f4b5408ab50 con 0x7f4b600d1d90 2024-03-18T14:43:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.761+0000 7f4b6572c640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4b4406bbd0 0x7f4b4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.762+0000 7f4b6572c640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4b4406bbd0 0x7f4b4406e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c036000 tx=0x7f4b5c03c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:06.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.766+0000 7f4b51ffb640 1 -- 172.21.15.6:0/1538357720 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4b54058d70 con 0x7f4b600d1d90 2024-03-18T14:43:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.949+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b6004f620 con 0x7f4b600d1d90 2024-03-18T14:43:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.950+0000 7f4b51ffb640 1 -- 172.21.15.6:0/1538357720 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+18658 (secure 0 0 0) 0x7f4b5405ce10 con 0x7f4b600d1d90 2024-03-18T14:43:06.951 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:06.952 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:06.352665+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4b4406bbd0 msgr2=0x7f4b4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4b4406bbd0 0x7f4b4406e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c036000 tx=0x7f4b5c03c040 comp rx=0 tx=0).stop 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b600d1d90 msgr2=0x7f4b601f4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b600d1d90 0x7f4b601f4f80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400c820 tx=0x7f4b5400cc20 comp rx=0 tx=0).stop 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 shutdown_connections 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4b4406bbd0 0x7f4b4406e090 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b6015bd40 0x7f4b601f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b600d1d90 0x7f4b601f4f80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 --2- 172.21.15.6:0/1538357720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b600cfcd0 0x7f4b600b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 >> 172.21.15.6:0/1538357720 conn(0x7f4b6009ae20 msgr2=0x7f4b60159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 shutdown_connections 2024-03-18T14:43:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:06.952+0000 7f4b67990640 1 -- 172.21.15.6:0/1538357720 wait complete. 2024-03-18T14:43:06.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:42:54.939648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:42:55.638120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-18T14:43:03.985735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:06.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.071+0000 7efe3ddd4640 1 Processor -- start 2024-03-18T14:43:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.071+0000 7efe3ddd4640 1 -- start start 2024-03-18T14:43:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe3ddd4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efe380ce9d0 0x7efe380cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe3ddd4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe380d29a0 con 0x7efe380d2510 2024-03-18T14:43:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe3ddd4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe380cfac0 con 0x7efe380d0730 2024-03-18T14:43:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe3ddd4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efe380cd8d0 con 0x7efe380ce9d0 2024-03-18T14:43:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe377fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efe380ce9d0 0x7efe380cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe377fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efe380ce9d0 0x7efe380cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53910/0 (socket says 172.21.15.6:53910) 2024-03-18T14:43:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe36ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efe380d2510 0x7efe380d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37484/0 (socket says 172.21.15.6:37484) 2024-03-18T14:43:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe377fe640 1 -- 172.21.15.6:0/56926410 learned_addr learned my addr 172.21.15.6:0/56926410 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.072+0000 7efe377fe640 1 -- 172.21.15.6:0/56926410 >> v1:172.21.15.92:6789/0 conn(0x7efe380d0730 legacy=0x7efe3815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.073+0000 7efe377fe640 1 -- 172.21.15.6:0/56926410 >> v1:172.21.15.6:6789/0 conn(0x7efe380d2510 legacy=0x7efe380d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.073+0000 7efe377fe640 1 -- 172.21.15.6:0/56926410 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe380cccc0 con 0x7efe380ce9d0 2024-03-18T14:43:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.073+0000 7efe377fe640 1 --2- 172.21.15.6:0/56926410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380ce9d0 0x7efe380cc650 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efe28009850 tx=0x7efe2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd89e9df5c8ecf4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.073+0000 7efe367fc640 1 -- 172.21.15.6:0/56926410 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2802d040 con 0x7efe380ce9d0 2024-03-18T14:43:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.073+0000 7efe367fc640 1 -- 172.21.15.6:0/56926410 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe28004230 con 0x7efe380ce9d0 2024-03-18T14:43:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.073+0000 7efe367fc640 1 -- 172.21.15.6:0/56926410 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28003610 con 0x7efe380ce9d0 2024-03-18T14:43:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.074+0000 7efe3ddd4640 1 -- 172.21.15.6:0/56926410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380ce9d0 msgr2=0x7efe380cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.074+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/56926410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380ce9d0 0x7efe380cc650 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efe28009850 tx=0x7efe2802b8f0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.074+0000 7efe3ddd4640 1 -- 172.21.15.6:0/56926410 shutdown_connections 2024-03-18T14:43:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.074+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/56926410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380ce9d0 0x7efe380cc650 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.074+0000 7efe3ddd4640 1 -- 172.21.15.6:0/56926410 >> 172.21.15.6:0/56926410 conn(0x7efe3808b0c0 msgr2=0x7efe380d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.074+0000 7efe3ddd4640 1 -- 172.21.15.6:0/56926410 shutdown_connections 2024-03-18T14:43:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.074+0000 7efe3ddd4640 1 -- 172.21.15.6:0/56926410 wait complete. 2024-03-18T14:43:07.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.075+0000 7efe3ddd4640 1 Processor -- start 2024-03-18T14:43:07.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.075+0000 7efe3ddd4640 1 -- start start 2024-03-18T14:43:07.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe3ddd4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe380ce9d0 0x7efe3806db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe3ddd4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efe380d0730 0x7efe380c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe3ddd4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380d2510 0x7efe38151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe3ddd4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efe380cd560 con 0x7efe380d2510 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe3ddd4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efe380cd280 con 0x7efe380ce9d0 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe3ddd4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efe380cedb0 con 0x7efe380d0730 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe37fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380d2510 0x7efe38151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe37fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380d2510 0x7efe38151ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53914/0 (socket says 172.21.15.6:53914) 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe37fff640 1 -- 172.21.15.6:0/1746770381 learned_addr learned my addr 172.21.15.6:0/1746770381 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe377fe640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe380ce9d0 0x7efe3806db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe36ffd640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efe380d0730 0x7efe380c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.076+0000 7efe37fff640 1 -- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efe380d0730 msgr2=0x7efe380c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe37fff640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efe380d0730 0x7efe380c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe37fff640 1 -- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe380ce9d0 msgr2=0x7efe3806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe37fff640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe380ce9d0 0x7efe3806db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe37fff640 1 -- 172.21.15.6:0/1746770381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe380c8d60 con 0x7efe380d2510 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe377fe640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe380ce9d0 0x7efe3806db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe36ffd640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efe380d0730 0x7efe380c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe37fff640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380d2510 0x7efe38151ab0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c700 tx=0x7efe3000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe2ffff640 1 -- 172.21.15.6:0/1746770381 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30016020 con 0x7efe380d2510 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe2ffff640 1 -- 172.21.15.6:0/1746770381 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe3000aa50 con 0x7efe380d2510 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe380c9040 con 0x7efe380d2510 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe2ffff640 1 -- 172.21.15.6:0/1746770381 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3000ce10 con 0x7efe380d2510 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.077+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe380c9600 con 0x7efe380d2510 2024-03-18T14:43:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.078+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe380751b0 con 0x7efe380d2510 2024-03-18T14:43:07.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.078+0000 7efe2ffff640 1 -- 172.21.15.6:0/1746770381 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efe30019020 con 0x7efe380d2510 2024-03-18T14:43:07.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.079+0000 7efe2ffff640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efe1806bbd0 0x7efe1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.079+0000 7efe2ffff640 1 -- 172.21.15.6:0/1746770381 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7efe3008b030 con 0x7efe380d2510 2024-03-18T14:43:07.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.079+0000 7efe377fe640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efe1806bbd0 0x7efe1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.080+0000 7efe377fe640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efe1806bbd0 0x7efe1806e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe28035000 tx=0x7efe28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.083+0000 7efe2ffff640 1 -- 172.21.15.6:0/1746770381 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efe30020020 con 0x7efe380d2510 2024-03-18T14:43:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.264+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efe380cfe00 con 0x7efe380d2510 2024-03-18T14:43:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.265+0000 7efe2ffff640 1 -- 172.21.15.6:0/1746770381 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7efe30059250 con 0x7efe380d2510 2024-03-18T14:43:07.267 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efe1806bbd0 msgr2=0x7efe1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efe1806bbd0 0x7efe1806e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe28035000 tx=0x7efe28008030 comp rx=0 tx=0).stop 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380d2510 msgr2=0x7efe38151ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380d2510 0x7efe38151ab0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c700 tx=0x7efe3000cbc0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 shutdown_connections 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efe1806bbd0 0x7efe1806e090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe380d2510 0x7efe38151ab0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efe380d0730 0x7efe380c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 --2- 172.21.15.6:0/1746770381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe380ce9d0 0x7efe3806db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 >> 172.21.15.6:0/1746770381 conn(0x7efe3808b0c0 msgr2=0x7efe3806fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 shutdown_connections 2024-03-18T14:43:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.267+0000 7efe3ddd4640 1 -- 172.21.15.6:0/1746770381 wait complete. 2024-03-18T14:43:07.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.383+0000 7efc171b9640 1 Processor -- start 2024-03-18T14:43:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.383+0000 7efc171b9640 1 -- start start 2024-03-18T14:43:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc171b9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc100d0730 0x7efc100cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc171b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc100cfac0 con 0x7efc100d2510 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc171b9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc100cccc0 con 0x7efc100ce9d0 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc171b9640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efc100cd8d0 con 0x7efc100d0730 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc0ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc100d0730 0x7efc100cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc15756640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efc100d2510 0x7efc1015b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37488/0 (socket says 172.21.15.6:37488) 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc15756640 1 -- 172.21.15.6:0/2149266446 learned_addr learned my addr 172.21.15.6:0/2149266446 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc0ffff640 1 -- 172.21.15.6:0/2149266446 >> v1:172.21.15.92:6789/0 conn(0x7efc100ce9d0 legacy=0x7efc100d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc0ffff640 1 -- 172.21.15.6:0/2149266446 >> v1:172.21.15.6:6789/0 conn(0x7efc100d2510 legacy=0x7efc1015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc0ffff640 1 -- 172.21.15.6:0/2149266446 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc100ccfa0 con 0x7efc100d0730 2024-03-18T14:43:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.384+0000 7efc0f7fe640 1 -- 172.21.15.6:0/2149266446 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862644855 0 0) 0x7efc100cfac0 con 0x7efc100d2510 2024-03-18T14:43:07.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.385+0000 7efc0ffff640 1 --2- 172.21.15.6:0/2149266446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cc650 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0097a0 tx=0x7efbfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4110a42452cf27be server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.385+0000 7efc0f7fe640 1 -- 172.21.15.6:0/2149266446 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc035020 con 0x7efc100d0730 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.385+0000 7efc0f7fe640 1 -- 172.21.15.6:0/2149266446 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbfc031830 con 0x7efc100d0730 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.385+0000 7efc0f7fe640 1 -- 172.21.15.6:0/2149266446 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc031c00 con 0x7efc100d0730 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.386+0000 7efc171b9640 1 -- 172.21.15.6:0/2149266446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 msgr2=0x7efc100cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.386+0000 7efc171b9640 1 --2- 172.21.15.6:0/2149266446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cc650 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0097a0 tx=0x7efbfc02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.386+0000 7efc171b9640 1 -- 172.21.15.6:0/2149266446 shutdown_connections 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.386+0000 7efc171b9640 1 --2- 172.21.15.6:0/2149266446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cc650 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.386+0000 7efc171b9640 1 -- 172.21.15.6:0/2149266446 >> 172.21.15.6:0/2149266446 conn(0x7efc1008b0c0 msgr2=0x7efc100d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.386+0000 7efc171b9640 1 -- 172.21.15.6:0/2149266446 shutdown_connections 2024-03-18T14:43:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.386+0000 7efc171b9640 1 -- 172.21.15.6:0/2149266446 wait complete. 2024-03-18T14:43:07.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.387+0000 7efc171b9640 1 Processor -- start 2024-03-18T14:43:07.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.387+0000 7efc171b9640 1 -- start start 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc171b9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc100ce9d0 0x7efc10151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc171b9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc171b9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc100d2510 0x7efc101f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc171b9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efc100d28f0 con 0x7efc100d0730 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc171b9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efc100cd560 con 0x7efc100d2510 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc171b9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efc100cedb0 con 0x7efc100ce9d0 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53930/0 (socket says 172.21.15.6:53930) 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 -- 172.21.15.6:0/169058835 learned_addr learned my addr 172.21.15.6:0/169058835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc14f55640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc100ce9d0 0x7efc10151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc15756640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc100d2510 0x7efc101f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 -- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc100ce9d0 msgr2=0x7efc10151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc100ce9d0 0x7efc10151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 -- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc100d2510 msgr2=0x7efc101f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc100d2510 0x7efc101f1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc0ffff640 1 -- 172.21.15.6:0/169058835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc100c99b0 con 0x7efc100d0730 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.388+0000 7efc14f55640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc100ce9d0 0x7efc10151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.389+0000 7efc0ffff640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cb2e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efbfc02bee0 tx=0x7efbfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.389+0000 7efc0d7fa640 1 -- 172.21.15.6:0/169058835 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc041070 con 0x7efc100d0730 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.389+0000 7efc0d7fa640 1 -- 172.21.15.6:0/169058835 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbfc03aa30 con 0x7efc100d0730 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.389+0000 7efc0d7fa640 1 -- 172.21.15.6:0/169058835 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc035040 con 0x7efc100d0730 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.389+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc100c9c90 con 0x7efc100d0730 2024-03-18T14:43:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.389+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc100ca250 con 0x7efc100d0730 2024-03-18T14:43:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.390+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc100751b0 con 0x7efc100d0730 2024-03-18T14:43:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.391+0000 7efc0d7fa640 1 -- 172.21.15.6:0/169058835 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efbfc038020 con 0x7efc100d0730 2024-03-18T14:43:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.391+0000 7efc0d7fa640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efbf406bbd0 0x7efbf406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.391+0000 7efc0d7fa640 1 -- 172.21.15.6:0/169058835 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7efbfc0ac860 con 0x7efc100d0730 2024-03-18T14:43:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.392+0000 7efc14f55640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efbf406bbd0 0x7efbf406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.392+0000 7efc14f55640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efbf406bbd0 0x7efbf406e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7efc04000fb0 tx=0x7efc04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.396+0000 7efc0d7fa640 1 -- 172.21.15.6:0/169058835 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efbfc07b040 con 0x7efc100d0730 2024-03-18T14:43:07.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.576+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efc100cfe00 con 0x7efc100d0730 2024-03-18T14:43:07.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.576+0000 7efc0d7fa640 1 -- 172.21.15.6:0/169058835 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7efbfc07a940 con 0x7efc100d0730 2024-03-18T14:43:07.578 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efbf406bbd0 msgr2=0x7efbf406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efbf406bbd0 0x7efbf406e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7efc04000fb0 tx=0x7efc04009040 comp rx=0 tx=0).stop 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 msgr2=0x7efc100cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cb2e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efbfc02bee0 tx=0x7efbfc0046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 shutdown_connections 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efbf406bbd0 0x7efbf406e090 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc100d2510 0x7efc101f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc100d0730 0x7efc100cb2e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 --2- 172.21.15.6:0/169058835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc100ce9d0 0x7efc10151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 >> 172.21.15.6:0/169058835 conn(0x7efc1008b0c0 msgr2=0x7efc100b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 shutdown_connections 2024-03-18T14:43:07.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.578+0000 7efc171b9640 1 -- 172.21.15.6:0/169058835 wait complete. 2024-03-18T14:43:07.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:07.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.698+0000 7f9e01675640 1 Processor -- start 2024-03-18T14:43:07.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.698+0000 7f9e01675640 1 -- start start 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.698+0000 7f9e01675640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9dfc0d3570 0x7f9dfc0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.698+0000 7f9e01675640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dfc0d39d0 con 0x7f9dfc0d0450 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.698+0000 7f9e01675640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dfc0d2fd0 con 0x7f9dfc15e030 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.698+0000 7f9e01675640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9dfc15e410 con 0x7f9dfc0d3570 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9dfb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9dfc0d0450 0x7f9dfc0d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37490/0 (socket says 172.21.15.6:37490) 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9dfb7fe640 1 -- 172.21.15.6:0/1801175337 learned_addr learned my addr 172.21.15.6:0/1801175337 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9df3fff640 1 --2- 172.21.15.6:0/1801175337 >> v2:172.21.15.92:3300/0 conn(0x7f9dfc0d3570 0x7f9dfc0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9dfa7fc640 1 -- 172.21.15.6:0/1801175337 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 388820656 0 0) 0x7f9dfc0d39d0 con 0x7f9dfc0d0450 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9dfa7fc640 1 -- 172.21.15.6:0/1801175337 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de8003520 con 0x7f9dfc0d0450 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9dfa7fc640 1 -- 172.21.15.6:0/1801175337 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560675739 0 0) 0x7f9dfc0d2fd0 con 0x7f9dfc15e030 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9dfa7fc640 1 -- 172.21.15.6:0/1801175337 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dfc0d39d0 con 0x7f9dfc15e030 2024-03-18T14:43:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9df3fff640 1 -- 172.21.15.6:0/1801175337 >> v1:172.21.15.92:6789/0 conn(0x7f9dfc15e030 legacy=0x7f9dfc0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9df3fff640 1 -- 172.21.15.6:0/1801175337 >> v1:172.21.15.6:6789/0 conn(0x7f9dfc0d0450 legacy=0x7f9dfc0d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.699+0000 7f9df3fff640 1 -- 172.21.15.6:0/1801175337 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dfc0d3290 con 0x7f9dfc0d3570 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.700+0000 7f9df3fff640 1 --2- 172.21.15.6:0/1801175337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b6140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9de40097a0 tx=0x7f9de402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf9815db55290961 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.700+0000 7f9dfa7fc640 1 -- 172.21.15.6:0/1801175337 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4034070 con 0x7f9dfc0d3570 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.700+0000 7f9dfa7fc640 1 -- 172.21.15.6:0/1801175337 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de4030610 con 0x7f9dfc0d3570 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.700+0000 7f9dfa7fc640 1 -- 172.21.15.6:0/1801175337 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40309c0 con 0x7f9dfc0d3570 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.701+0000 7f9e01675640 1 -- 172.21.15.6:0/1801175337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d3570 msgr2=0x7f9dfc0b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.701+0000 7f9e01675640 1 --2- 172.21.15.6:0/1801175337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b6140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9de40097a0 tx=0x7f9de402b960 comp rx=0 tx=0).stop 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.701+0000 7f9e01675640 1 -- 172.21.15.6:0/1801175337 shutdown_connections 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.701+0000 7f9e01675640 1 --2- 172.21.15.6:0/1801175337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b6140 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.701+0000 7f9e01675640 1 -- 172.21.15.6:0/1801175337 >> 172.21.15.6:0/1801175337 conn(0x7f9dfc06c8c0 msgr2=0x7f9dfc06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.701+0000 7f9e01675640 1 -- 172.21.15.6:0/1801175337 shutdown_connections 2024-03-18T14:43:07.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.701+0000 7f9e01675640 1 -- 172.21.15.6:0/1801175337 wait complete. 2024-03-18T14:43:07.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.702+0000 7f9e01675640 1 Processor -- start 2024-03-18T14:43:07.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.703+0000 7f9e01675640 1 -- start start 2024-03-18T14:43:07.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9e01675640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d0450 0x7f9dfc0c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9e01675640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9e01675640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dfc15e030 0x7f9dfc0b83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9e01675640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc0ce830 con 0x7f9dfc0d3570 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9e01675640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc0ce300 con 0x7f9dfc0d0450 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9e01675640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9dfc161a90 con 0x7f9dfc15e030 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53934/0 (socket says 172.21.15.6:53934) 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 -- 172.21.15.6:0/1190554836 learned_addr learned my addr 172.21.15.6:0/1190554836 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9dfb7fe640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dfc15e030 0x7f9dfc0b83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9dfaffd640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d0450 0x7f9dfc0c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 -- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dfc15e030 msgr2=0x7f9dfc0b83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dfc15e030 0x7f9dfc0b83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 -- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d0450 msgr2=0x7f9dfc0c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d0450 0x7f9dfc0c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 -- 172.21.15.6:0/1190554836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dfc0c9d70 con 0x7f9dfc0d3570 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.704+0000 7f9df3fff640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b7e60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9de4000c00 tx=0x7f9de40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.705+0000 7f9df37fe640 1 -- 172.21.15.6:0/1190554836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4034040 con 0x7f9dfc0d3570 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.705+0000 7f9df37fe640 1 -- 172.21.15.6:0/1190554836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de4030410 con 0x7f9dfc0d3570 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.705+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dfc147ad0 con 0x7f9dfc0d3570 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.705+0000 7f9df37fe640 1 -- 172.21.15.6:0/1190554836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4006e10 con 0x7f9dfc0d3570 2024-03-18T14:43:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.705+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dfc148090 con 0x7f9dfc0d3570 2024-03-18T14:43:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.706+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dfc0751b0 con 0x7f9dfc0d3570 2024-03-18T14:43:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.706+0000 7f9df37fe640 1 -- 172.21.15.6:0/1190554836 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9de4037070 con 0x7f9dfc0d3570 2024-03-18T14:43:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.706+0000 7f9df37fe640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ddc06bbd0 0x7f9ddc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.707+0000 7f9df37fe640 1 -- 172.21.15.6:0/1190554836 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f9de40acad0 con 0x7f9dfc0d3570 2024-03-18T14:43:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.707+0000 7f9dfaffd640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ddc06bbd0 0x7f9ddc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.707+0000 7f9dfaffd640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ddc06bbd0 0x7f9ddc06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9dec007fd0 tx=0x7f9dec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:07.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.715+0000 7f9df37fe640 1 -- 172.21.15.6:0/1190554836 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9dfc0c9d70 con 0x7f9dfc0d3570 2024-03-18T14:43:07.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.899+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9dfc0d0ab0 con 0x7f9dfc0d3570 2024-03-18T14:43:07.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.900+0000 7f9df37fe640 1 -- 172.21.15.6:0/1190554836 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f9de40305e0 con 0x7f9dfc0d3570 2024-03-18T14:43:07.901 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ddc06bbd0 msgr2=0x7f9ddc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ddc06bbd0 0x7f9ddc06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9dec007fd0 tx=0x7f9dec009040 comp rx=0 tx=0).stop 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dfc0d3570 msgr2=0x7f9dfc0b7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b7e60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9de4000c00 tx=0x7f9de40046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 shutdown_connections 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ddc06bbd0 0x7f9ddc06e090 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dfc15e030 0x7f9dfc0b83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dfc0d3570 0x7f9dfc0b7e60 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 --2- 172.21.15.6:0/1190554836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dfc0d0450 0x7f9dfc0c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:07.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 >> 172.21.15.6:0/1190554836 conn(0x7f9dfc06c8c0 msgr2=0x7f9dfc0ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:07.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 shutdown_connections 2024-03-18T14:43:07.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:07.902+0000 7f9e01675640 1 -- 172.21.15.6:0/1190554836 wait complete. 2024-03-18T14:43:07.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-18T14:43:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.018+0000 7fbf00d3b640 1 Processor -- start 2024-03-18T14:43:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.018+0000 7fbf00d3b640 1 -- start start 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbf00d3b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbefc06b430 0x7fbefc0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbf00d3b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbf00d3b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbefc15d640 0x7fbefc15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbf00d3b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbefc15e650 con 0x7fbefc15d640 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbf00d3b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbefc15e930 con 0x7fbefc06b430 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbf00d3b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbefc15ec10 con 0x7fbefc06b810 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbefc15d640 0x7fbefc15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbefc15d640 0x7fbefc15dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53950/0 (socket says 172.21.15.6:53950) 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 -- 172.21.15.6:0/3717173861 learned_addr learned my addr 172.21.15.6:0/3717173861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 -- 172.21.15.6:0/3717173861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 msgr2=0x7fbefc0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbef9d9b640 1 --2- 172.21.15.6:0/3717173861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefa59c640 1 --2- 172.21.15.6:0/3717173861 >> v2:172.21.15.92:3300/0 conn(0x7fbefc06b430 0x7fbefc0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 --2- 172.21.15.6:0/3717173861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 -- 172.21.15.6:0/3717173861 >> v2:172.21.15.92:3300/0 conn(0x7fbefc06b430 msgr2=0x7fbefc0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 --2- 172.21.15.6:0/3717173861 >> v2:172.21.15.92:3300/0 conn(0x7fbefc06b430 0x7fbefc0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefad9d640 1 -- 172.21.15.6:0/3717173861 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc15eef0 con 0x7fbefc15d640 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbefa59c640 1 --2- 172.21.15.6:0/3717173861 >> v2:172.21.15.92:3300/0 conn(0x7fbefc06b430 0x7fbefc0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.019+0000 7fbef9d9b640 1 --2- 172.21.15.6:0/3717173861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.020+0000 7fbefad9d640 1 --2- 172.21.15.6:0/3717173861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc15d640 0x7fbefc15dac0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbefc06c2a0 tx=0x7fbef402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a5b36e74bb3ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.020+0000 7fbef959a640 1 -- 172.21.15.6:0/3717173861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef402ca70 con 0x7fbefc15d640 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.020+0000 7fbef959a640 1 -- 172.21.15.6:0/3717173861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef402cc10 con 0x7fbefc15d640 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.020+0000 7fbef959a640 1 -- 172.21.15.6:0/3717173861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef40343f0 con 0x7fbefc15d640 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 -- 172.21.15.6:0/3717173861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc15d640 msgr2=0x7fbefc15dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/3717173861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc15d640 0x7fbefc15dac0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbefc06c2a0 tx=0x7fbef402bed0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 -- 172.21.15.6:0/3717173861 shutdown_connections 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/3717173861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc15d640 0x7fbefc15dac0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/3717173861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/3717173861 >> v2:172.21.15.92:3300/0 conn(0x7fbefc06b430 0x7fbefc0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 -- 172.21.15.6:0/3717173861 >> 172.21.15.6:0/3717173861 conn(0x7fbefc099a50 msgr2=0x7fbefc0d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 -- 172.21.15.6:0/3717173861 shutdown_connections 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.021+0000 7fbf00d3b640 1 -- 172.21.15.6:0/3717173861 wait complete. 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.022+0000 7fbf00d3b640 1 Processor -- start 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.022+0000 7fbf00d3b640 1 -- start start 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbf00d3b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc06b430 0x7fbefc157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbf00d3b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbf00d3b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc15d640 0x7fbefc0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbf00d3b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc0b20f0 con 0x7fbefc06b430 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbf00d3b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc0b2270 con 0x7fbefc15d640 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbf00d3b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbefc15e000 con 0x7fbefc06b810 2024-03-18T14:43:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc06b430 0x7fbefc157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc06b430 0x7fbefc157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53964/0 (socket says 172.21.15.6:53964) 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 -- 172.21.15.6:0/1302713834 learned_addr learned my addr 172.21.15.6:0/1302713834 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbef9d9b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefad9d640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc15d640 0x7fbefc0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 -- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 msgr2=0x7fbefc0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 -- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc15d640 msgr2=0x7fbefc0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc15d640 0x7fbefc0ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 -- 172.21.15.6:0/1302713834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc0ae860 con 0x7fbefc06b430 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbefa59c640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc06b430 0x7fbefc157ba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c7f0 tx=0x7fbeec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbeeaffd640 1 -- 172.21.15.6:0/1302713834 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec015070 con 0x7fbefc06b430 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.023+0000 7fbeeaffd640 1 -- 172.21.15.6:0/1302713834 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeec007a30 con 0x7fbefc06b430 2024-03-18T14:43:08.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.024+0000 7fbeeaffd640 1 -- 172.21.15.6:0/1302713834 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec007e00 con 0x7fbefc06b430 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.024+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbefc0aeb40 con 0x7fbefc06b430 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.024+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbefc0af100 con 0x7fbefc06b430 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.024+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbefc0751b0 con 0x7fbefc06b430 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.025+0000 7fbeeaffd640 1 -- 172.21.15.6:0/1302713834 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbeec018070 con 0x7fbefc06b430 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.025+0000 7fbeeaffd640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.026+0000 7fbeeaffd640 1 -- 172.21.15.6:0/1302713834 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fbeec08ac70 con 0x7fbefc06b430 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.026+0000 7fbef9d9b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.026+0000 7fbef9d9b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0c4830 tx=0x7fbef0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.030+0000 7fbeeaffd640 1 -- 172.21.15.6:0/1302713834 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbeec0594e0 con 0x7fbefc06b430 2024-03-18T14:43:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.344+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fbefc0d1380 con 0x7fbefc06b430 2024-03-18T14:43:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.344+0000 7fbeeaffd640 1 -- 172.21.15.6:0/1302713834 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) ==== 97+0+11 (secure 0 0 0) 0x7fbeec058de0 con 0x7fbefc06b430 2024-03-18T14:43:08.346 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.346+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 msgr2=0x7fbedc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.346+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0c4830 tx=0x7fbef0008040 comp rx=0 tx=0).stop 2024-03-18T14:43:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc06b430 msgr2=0x7fbefc157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc06b430 0x7fbefc157ba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c7f0 tx=0x7fbeec00ccb0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 shutdown_connections 2024-03-18T14:43:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbedc06bbd0 0x7fbedc06e090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbefc15d640 0x7fbefc0ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbefc06b810 0x7fbefc0c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 --2- 172.21.15.6:0/1302713834 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbefc06b430 0x7fbefc157ba0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 >> 172.21.15.6:0/1302713834 conn(0x7fbefc099a50 msgr2=0x7fbefc0b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 shutdown_connections 2024-03-18T14:43:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.347+0000 7fbf00d3b640 1 -- 172.21.15.6:0/1302713834 wait complete. 2024-03-18T14:43:08.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:08.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.460+0000 7f44de034640 1 Processor -- start 2024-03-18T14:43:08.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.461+0000 7f44de034640 1 -- start start 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44de034640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f44d815ac20 0x7f44d80ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44de034640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44d806b8d0 0x7f44d815e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44de034640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44d80ce650 con 0x7f44d80cf6f0 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44de034640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f44d80b4ba0 con 0x7f44d806b8d0 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44de034640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f44d80ce930 con 0x7f44d815ac20 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d7fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44d806b8d0 0x7f44d815e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d6ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44d80cf6f0 0x7f44d806b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37498/0 (socket says 172.21.15.6:37498) 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d6ffd640 1 -- 172.21.15.6:0/311205472 learned_addr learned my addr 172.21.15.6:0/311205472 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d77fe640 1 --2- 172.21.15.6:0/311205472 >> v2:172.21.15.92:3300/0 conn(0x7f44d815ac20 0x7f44d80ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d7fff640 1 -- 172.21.15.6:0/311205472 >> v2:172.21.15.92:3300/0 conn(0x7f44d815ac20 msgr2=0x7f44d80ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d7fff640 1 --2- 172.21.15.6:0/311205472 >> v2:172.21.15.92:3300/0 conn(0x7f44d815ac20 0x7f44d80ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d7fff640 1 -- 172.21.15.6:0/311205472 >> v1:172.21.15.6:6789/0 conn(0x7f44d80cf6f0 legacy=0x7f44d806b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d67fc640 1 -- 172.21.15.6:0/311205472 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1134861270 0 0) 0x7f44d80ce650 con 0x7f44d80cf6f0 2024-03-18T14:43:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d7fff640 1 -- 172.21.15.6:0/311205472 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d80cec10 con 0x7f44d806b8d0 2024-03-18T14:43:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.462+0000 7f44d77fe640 1 --2- 172.21.15.6:0/311205472 >> v2:172.21.15.92:3300/0 conn(0x7f44d815ac20 0x7f44d80ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.463+0000 7f44d7fff640 1 --2- 172.21.15.6:0/311205472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d806b8d0 0x7f44d815e6d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f44d8071c20 tx=0x7f44d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81472cf4ca966c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44d67fc640 1 -- 172.21.15.6:0/311205472 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0039070 con 0x7f44d806b8d0 2024-03-18T14:43:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44d67fc640 1 -- 172.21.15.6:0/311205472 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d002cc80 con 0x7f44d806b8d0 2024-03-18T14:43:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44d67fc640 1 -- 172.21.15.6:0/311205472 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0034660 con 0x7f44d806b8d0 2024-03-18T14:43:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 -- 172.21.15.6:0/311205472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d806b8d0 msgr2=0x7f44d815e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 --2- 172.21.15.6:0/311205472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d806b8d0 0x7f44d815e6d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f44d8071c20 tx=0x7f44d002bed0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 -- 172.21.15.6:0/311205472 shutdown_connections 2024-03-18T14:43:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 --2- 172.21.15.6:0/311205472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d806b8d0 0x7f44d815e6d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 --2- 172.21.15.6:0/311205472 >> v2:172.21.15.92:3300/0 conn(0x7f44d815ac20 0x7f44d80ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 -- 172.21.15.6:0/311205472 >> 172.21.15.6:0/311205472 conn(0x7f44d806e280 msgr2=0x7f44d806e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 -- 172.21.15.6:0/311205472 shutdown_connections 2024-03-18T14:43:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.464+0000 7f44de034640 1 -- 172.21.15.6:0/311205472 wait complete. 2024-03-18T14:43:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.465+0000 7f44de034640 1 Processor -- start 2024-03-18T14:43:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44de034640 1 -- start start 2024-03-18T14:43:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44de034640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44d806b8d0 0x7f44d8142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44de034640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d80cf6f0 0x7f44d81e5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44de034640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44d815ac20 0x7f44d81e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44de034640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f44d80ccff0 con 0x7f44d80cf6f0 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44de034640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f44d80cf1d0 con 0x7f44d815ac20 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44de034640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f44d80cfb80 con 0x7f44d806b8d0 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44d6ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d80cf6f0 0x7f44d81e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44d6ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d80cf6f0 0x7f44d81e5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53978/0 (socket says 172.21.15.6:53978) 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.466+0000 7f44d6ffd640 1 -- 172.21.15.6:0/3018901294 learned_addr learned my addr 172.21.15.6:0/3018901294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d77fe640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44d806b8d0 0x7f44d8142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d7fff640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44d815ac20 0x7f44d81e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d6ffd640 1 -- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44d806b8d0 msgr2=0x7f44d8142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d6ffd640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44d806b8d0 0x7f44d8142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d6ffd640 1 -- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44d815ac20 msgr2=0x7f44d81e5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d6ffd640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44d815ac20 0x7f44d81e5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d6ffd640 1 -- 172.21.15.6:0/3018901294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d81e4760 con 0x7f44d80cf6f0 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d77fe640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44d806b8d0 0x7f44d8142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d7fff640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44d815ac20 0x7f44d81e5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44d6ffd640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d80cf6f0 0x7f44d81e5520 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00c4d0 tx=0x7f44cc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44bffff640 1 -- 172.21.15.6:0/3018901294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc018070 con 0x7f44d80cf6f0 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44bffff640 1 -- 172.21.15.6:0/3018901294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44cc007500 con 0x7f44d80cf6f0 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d81e4a40 con 0x7f44d80cf6f0 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44bffff640 1 -- 172.21.15.6:0/3018901294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc004a20 con 0x7f44d80cf6f0 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.467+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d81e5000 con 0x7f44d80cf6f0 2024-03-18T14:43:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.468+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d81e5200 con 0x7f44d80cf6f0 2024-03-18T14:43:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.469+0000 7f44bffff640 1 -- 172.21.15.6:0/3018901294 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f44cc01b070 con 0x7f44d80cf6f0 2024-03-18T14:43:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.469+0000 7f44bffff640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f44c406bbd0 0x7f44c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.469+0000 7f44bffff640 1 -- 172.21.15.6:0/3018901294 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f44cc08a0e0 con 0x7f44d80cf6f0 2024-03-18T14:43:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.469+0000 7f44d77fe640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f44c406bbd0 0x7f44c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.470+0000 7f44d77fe640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f44c406bbd0 0x7f44c406e090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f44c800a990 tx=0x7f44c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.474+0000 7f44bffff640 1 -- 172.21.15.6:0/3018901294 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f44d81e5200 con 0x7f44d80cf6f0 2024-03-18T14:43:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.708+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f44d804f5b0 con 0x7f44d80cf6f0 2024-03-18T14:43:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.708+0000 7f44bffff640 1 -- 172.21.15.6:0/3018901294 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) ==== 80+0+0 (secure 0 0 0) 0x7f44cc0588c0 con 0x7f44d80cf6f0 2024-03-18T14:43:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f44c406bbd0 msgr2=0x7f44c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f44c406bbd0 0x7f44c406e090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f44c800a990 tx=0x7f44c8008040 comp rx=0 tx=0).stop 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d80cf6f0 msgr2=0x7f44d81e5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d80cf6f0 0x7f44d81e5520 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00c4d0 tx=0x7f44cc00c990 comp rx=0 tx=0).stop 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 shutdown_connections 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f44c406bbd0 0x7f44c406e090 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44d815ac20 0x7f44d81e5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d80cf6f0 0x7f44d81e5520 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 --2- 172.21.15.6:0/3018901294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44d806b8d0 0x7f44d8142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 >> 172.21.15.6:0/3018901294 conn(0x7f44d806e280 msgr2=0x7f44d806b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 shutdown_connections 2024-03-18T14:43:08.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.711+0000 7f44de034640 1 -- 172.21.15.6:0/3018901294 wait complete. 2024-03-18T14:43:08.724 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:08.724 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:08.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:08.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.878+0000 7efc592ff640 1 Processor -- start 2024-03-18T14:43:08.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.878+0000 7efc592ff640 1 -- start start 2024-03-18T14:43:08.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc592ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc540b9650 0x7efc540d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc592ff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc540d0ff0 con 0x7efc5415e170 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc592ff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc540b9ae0 con 0x7efc540d0bc0 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc592ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efc540cef50 con 0x7efc540b9650 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc527fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efc540b9650 0x7efc540d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc52ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efc5415e170 0x7efc540d3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37500/0 (socket says 172.21.15.6:37500) 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc52ffd640 1 -- 172.21.15.6:0/2285819212 learned_addr learned my addr 172.21.15.6:0/2285819212 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc527fc640 1 -- 172.21.15.6:0/2285819212 >> v1:172.21.15.92:6789/0 conn(0x7efc540d0bc0 legacy=0x7efc540b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc527fc640 1 -- 172.21.15.6:0/2285819212 >> v1:172.21.15.6:6789/0 conn(0x7efc5415e170 legacy=0x7efc540d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc51ffb640 1 -- 172.21.15.6:0/2285819212 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 289422687 0 0) 0x7efc540d0ff0 con 0x7efc5415e170 2024-03-18T14:43:08.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.879+0000 7efc527fc640 1 -- 172.21.15.6:0/2285819212 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc540cf230 con 0x7efc540b9650 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.880+0000 7efc527fc640 1 --2- 172.21.15.6:0/2285819212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540b9650 0x7efc540d0680 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7efc480097a0 tx=0x7efc4802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb28cf6dfc18fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.880+0000 7efc51ffb640 1 -- 172.21.15.6:0/2285819212 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48034070 con 0x7efc540b9650 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc51ffb640 1 -- 172.21.15.6:0/2285819212 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc48030610 con 0x7efc540b9650 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc51ffb640 1 -- 172.21.15.6:0/2285819212 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc480309e0 con 0x7efc540b9650 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc592ff640 1 -- 172.21.15.6:0/2285819212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540b9650 msgr2=0x7efc540d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc592ff640 1 --2- 172.21.15.6:0/2285819212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540b9650 0x7efc540d0680 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7efc480097a0 tx=0x7efc4802b960 comp rx=0 tx=0).stop 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc592ff640 1 -- 172.21.15.6:0/2285819212 shutdown_connections 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc592ff640 1 --2- 172.21.15.6:0/2285819212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540b9650 0x7efc540d0680 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc592ff640 1 -- 172.21.15.6:0/2285819212 >> 172.21.15.6:0/2285819212 conn(0x7efc5406c900 msgr2=0x7efc5406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc592ff640 1 -- 172.21.15.6:0/2285819212 shutdown_connections 2024-03-18T14:43:08.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.881+0000 7efc592ff640 1 -- 172.21.15.6:0/2285819212 wait complete. 2024-03-18T14:43:08.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 Processor -- start 2024-03-18T14:43:08.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 -- start start 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc540b9650 0x7efc541e8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540d0bc0 0x7efc5415b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc5415e170 0x7efc54148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efc540d3a40 con 0x7efc540d0bc0 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efc540d2e10 con 0x7efc5415e170 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.883+0000 7efc592ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7efc540d2820 con 0x7efc540b9650 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540d0bc0 0x7efc5415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540d0bc0 0x7efc5415b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54008/0 (socket says 172.21.15.6:54008) 2024-03-18T14:43:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 -- 172.21.15.6:0/3498734114 learned_addr learned my addr 172.21.15.6:0/3498734114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc52ffd640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc540b9650 0x7efc541e8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc537fe640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc5415e170 0x7efc54148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 -- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc540b9650 msgr2=0x7efc541e8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc540b9650 0x7efc541e8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 -- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc5415e170 msgr2=0x7efc54148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc5415e170 0x7efc54148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 -- 172.21.15.6:0/3498734114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc541e9070 con 0x7efc540d0bc0 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc537fe640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc5415e170 0x7efc54148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc527fc640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540d0bc0 0x7efc5415b940 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7efc480097a0 tx=0x7efc480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc3b7fe640 1 -- 172.21.15.6:0/3498734114 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48034040 con 0x7efc540d0bc0 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc3b7fe640 1 -- 172.21.15.6:0/3498734114 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc48004030 con 0x7efc540d0bc0 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc3b7fe640 1 -- 172.21.15.6:0/3498734114 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48004420 con 0x7efc540d0bc0 2024-03-18T14:43:08.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc5415a8a0 con 0x7efc540d0bc0 2024-03-18T14:43:08.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.884+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc5415ae60 con 0x7efc540d0bc0 2024-03-18T14:43:08.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.885+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc5408afe0 con 0x7efc540d0bc0 2024-03-18T14:43:08.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.886+0000 7efc3b7fe640 1 -- 172.21.15.6:0/3498734114 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efc48037070 con 0x7efc540d0bc0 2024-03-18T14:43:08.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.887+0000 7efc3b7fe640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc4006bc00 0x7efc4006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:08.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.887+0000 7efc3b7fe640 1 -- 172.21.15.6:0/3498734114 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7efc480aca60 con 0x7efc540d0bc0 2024-03-18T14:43:08.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.887+0000 7efc52ffd640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc4006bc00 0x7efc4006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:08.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.887+0000 7efc52ffd640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc4006bc00 0x7efc4006e0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f60 tx=0x7efc44007d00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:08.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:08.895+0000 7efc3b7fe640 1 -- 172.21.15.6:0/3498734114 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efc4807b2e0 con 0x7efc540d0bc0 2024-03-18T14:43:09.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.077+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc540b38e0 con 0x7efc540d0bc0 2024-03-18T14:43:09.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.079+0000 7efc3b7fe640 1 -- 172.21.15.6:0/3498734114 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+18658 (secure 0 0 0) 0x7efc4807abe0 con 0x7efc540d0bc0 2024-03-18T14:43:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:09.080 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:06.352665+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc4006bc00 msgr2=0x7efc4006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc4006bc00 0x7efc4006e0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f60 tx=0x7efc44007d00 comp rx=0 tx=0).stop 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540d0bc0 msgr2=0x7efc5415b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540d0bc0 0x7efc5415b940 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7efc480097a0 tx=0x7efc480046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 shutdown_connections 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7efc4006bc00 0x7efc4006e0c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc5415e170 0x7efc54148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc540d0bc0 0x7efc5415b940 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 --2- 172.21.15.6:0/3498734114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc540b9650 0x7efc541e8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 >> 172.21.15.6:0/3498734114 conn(0x7efc5406c900 msgr2=0x7efc540b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:09.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 shutdown_connections 2024-03-18T14:43:09.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.081+0000 7efc592ff640 1 -- 172.21.15.6:0/3498734114 wait complete. 2024-03-18T14:43:09.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:42:54.939648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:42:55.638120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-18T14:43:03.985735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:09.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:09.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.201+0000 7f74c6b76640 1 Processor -- start 2024-03-18T14:43:09.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.201+0000 7f74c6b76640 1 -- start start 2024-03-18T14:43:09.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c6b76640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74c00ce930 0x7f74c006cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c6b76640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00d0850 0x7f74c015d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c6b76640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74c00d0f30 con 0x7f74c006d020 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c6b76640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f74c00d0ce0 con 0x7f74c00ce930 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c6b76640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f74c00b1070 con 0x7f74c00d0850 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c4912640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74c00ce930 0x7f74c006cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c4912640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74c00ce930 0x7f74c006cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54014/0 (socket says 172.21.15.6:54014) 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c4912640 1 -- 172.21.15.6:0/90368494 learned_addr learned my addr 172.21.15.6:0/90368494 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74bffff640 1 --1- 172.21.15.6:0/90368494 >> v1:172.21.15.6:6789/0 conn(0x7f74c006d020 0x7f74c00d0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37516/0 (socket says 172.21.15.6:37516) 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c5113640 1 --2- 172.21.15.6:0/90368494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00d0850 0x7f74c015d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c4912640 1 -- 172.21.15.6:0/90368494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00d0850 msgr2=0x7f74c015d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c4912640 1 --2- 172.21.15.6:0/90368494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00d0850 0x7f74c015d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c4912640 1 -- 172.21.15.6:0/90368494 >> v1:172.21.15.6:6789/0 conn(0x7f74c006d020 legacy=0x7f74c00d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c4912640 1 -- 172.21.15.6:0/90368494 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c00b1350 con 0x7f74c00ce930 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74bf7fe640 1 -- 172.21.15.6:0/90368494 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592802397 0 0) 0x7f74c00d0f30 con 0x7f74c006d020 2024-03-18T14:43:09.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.202+0000 7f74c5113640 1 --2- 172.21.15.6:0/90368494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00d0850 0x7f74c015d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:09.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.203+0000 7f74c4912640 1 --2- 172.21.15.6:0/90368494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00ce930 0x7f74c006cae0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009850 tx=0x7f74b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf8dd544f345166 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:09.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.203+0000 7f74bf7fe640 1 -- 172.21.15.6:0/90368494 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4004230 con 0x7f74c00ce930 2024-03-18T14:43:09.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.203+0000 7f74bf7fe640 1 -- 172.21.15.6:0/90368494 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b40043d0 con 0x7f74c00ce930 2024-03-18T14:43:09.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.203+0000 7f74bf7fe640 1 -- 172.21.15.6:0/90368494 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4003810 con 0x7f74c00ce930 2024-03-18T14:43:09.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 -- 172.21.15.6:0/90368494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00ce930 msgr2=0x7f74c006cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 --2- 172.21.15.6:0/90368494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00ce930 0x7f74c006cae0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009850 tx=0x7f74b402b960 comp rx=0 tx=0).stop 2024-03-18T14:43:09.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 -- 172.21.15.6:0/90368494 shutdown_connections 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 --2- 172.21.15.6:0/90368494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00d0850 0x7f74c015d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 --2- 172.21.15.6:0/90368494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00ce930 0x7f74c006cae0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 -- 172.21.15.6:0/90368494 >> 172.21.15.6:0/90368494 conn(0x7f74c008b0c0 msgr2=0x7f74c00d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 -- 172.21.15.6:0/90368494 shutdown_connections 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.204+0000 7f74c6b76640 1 -- 172.21.15.6:0/90368494 wait complete. 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.205+0000 7f74c6b76640 1 Processor -- start 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.205+0000 7f74c6b76640 1 -- start start 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c6b76640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006d020 0x7f74c01f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c6b76640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00ce930 0x7f74c01e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c6b76640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00d0850 0x7f74c01e8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c6b76640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f74c015dd70 con 0x7f74c00d0850 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c6b76640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74c015dbf0 con 0x7f74c006d020 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c6b76640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f74c015def0 con 0x7f74c00ce930 2024-03-18T14:43:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00d0850 0x7f74c01e8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00d0850 0x7f74c01e8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54022/0 (socket says 172.21.15.6:54022) 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 -- 172.21.15.6:0/2726326282 learned_addr learned my addr 172.21.15.6:0/2726326282 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74bffff640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00ce930 0x7f74c01e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c4912640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006d020 0x7f74c01f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 -- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00ce930 msgr2=0x7f74c01e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00ce930 0x7f74c01e8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 -- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006d020 msgr2=0x7f74c01f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006d020 0x7f74c01f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 -- 172.21.15.6:0/2726326282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c01e9410 con 0x7f74c00d0850 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74c5113640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00d0850 0x7f74c01e8840 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f74b800cd80 tx=0x7f74b800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.206+0000 7f74bd7fa640 1 -- 172.21.15.6:0/2726326282 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b800f680 con 0x7f74c00d0850 2024-03-18T14:43:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.207+0000 7f74bd7fa640 1 -- 172.21.15.6:0/2726326282 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b800f840 con 0x7f74c00d0850 2024-03-18T14:43:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.207+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c01e8e50 con 0x7f74c00d0850 2024-03-18T14:43:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.207+0000 7f74bd7fa640 1 -- 172.21.15.6:0/2726326282 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b800fc30 con 0x7f74c00d0850 2024-03-18T14:43:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.207+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c00b5590 con 0x7f74c00d0850 2024-03-18T14:43:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.207+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c00751b0 con 0x7f74c00d0850 2024-03-18T14:43:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.208+0000 7f74bd7fa640 1 -- 172.21.15.6:0/2726326282 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f74b80213f0 con 0x7f74c00d0850 2024-03-18T14:43:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.208+0000 7f74bd7fa640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f74a406bc00 0x7f74a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.209+0000 7f74bd7fa640 1 -- 172.21.15.6:0/2726326282 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f74b808b9d0 con 0x7f74c00d0850 2024-03-18T14:43:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.209+0000 7f74c4912640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f74a406bc00 0x7f74a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.209+0000 7f74c4912640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f74a406bc00 0x7f74a406e0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f74b4035000 tx=0x7f74b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:09.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.213+0000 7f74bd7fa640 1 -- 172.21.15.6:0/2726326282 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f74b8014030 con 0x7f74c00d0850 2024-03-18T14:43:09.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.522+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74c00b0db0 con 0x7f74c00d0850 2024-03-18T14:43:09.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.522+0000 7f74bd7fa640 1 -- 172.21.15.6:0/2726326282 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f74b8059b60 con 0x7f74c00d0850 2024-03-18T14:43:09.524 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f74a406bc00 msgr2=0x7f74a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f74a406bc00 0x7f74a406e0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f74b4035000 tx=0x7f74b4008030 comp rx=0 tx=0).stop 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00d0850 msgr2=0x7f74c01e8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00d0850 0x7f74c01e8840 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f74b800cd80 tx=0x7f74b800a650 comp rx=0 tx=0).stop 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 shutdown_connections 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f74a406bc00 0x7f74a406e0c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c00d0850 0x7f74c01e8840 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c00ce930 0x7f74c01e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 --2- 172.21.15.6:0/2726326282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006d020 0x7f74c01f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 >> 172.21.15.6:0/2726326282 conn(0x7f74c008b0c0 msgr2=0x7f74c006fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:09.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 shutdown_connections 2024-03-18T14:43:09.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.524+0000 7f74c6b76640 1 -- 172.21.15.6:0/2726326282 wait complete. 2024-03-18T14:43:09.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.640+0000 7fd3ed5b3640 1 Processor -- start 2024-03-18T14:43:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.640+0000 7fd3ed5b3640 1 -- start start 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3ed5b3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e80ce930 0x7fd3e806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3ed5b3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3e806d020 0x7fd3e80d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3ed5b3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e80d0f30 con 0x7fd3e80d0680 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3ed5b3640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e806d480 con 0x7fd3e806d020 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3ed5b3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e80b1070 con 0x7fd3e80ce930 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3e67fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3e806d020 0x7fd3e80d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3e77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd3e80d0680 0x7fd3e815d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37520/0 (socket says 172.21.15.6:37520) 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3e67fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd3e806d020 0x7fd3e80d0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54036/0 (socket says 172.21.15.6:54036) 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3e77fe640 1 -- 172.21.15.6:0/1413687641 learned_addr learned my addr 172.21.15.6:0/1413687641 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.641+0000 7fd3e6ffd640 1 --2- 172.21.15.6:0/1413687641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e80ce930 0x7fd3e806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.642+0000 7fd3e67fc640 1 -- 172.21.15.6:0/1413687641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e80ce930 msgr2=0x7fd3e806cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:09.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.642+0000 7fd3e67fc640 1 --2- 172.21.15.6:0/1413687641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e80ce930 0x7fd3e806cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.642+0000 7fd3e67fc640 1 -- 172.21.15.6:0/1413687641 >> v1:172.21.15.6:6789/0 conn(0x7fd3e80d0680 legacy=0x7fd3e815d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.642+0000 7fd3e67fc640 1 -- 172.21.15.6:0/1413687641 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e80b1350 con 0x7fd3e806d020 2024-03-18T14:43:09.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.642+0000 7fd3e5ffb640 1 -- 172.21.15.6:0/1413687641 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4187966836 0 0) 0x7fd3e80d0f30 con 0x7fd3e80d0680 2024-03-18T14:43:09.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.642+0000 7fd3e67fc640 1 --2- 172.21.15.6:0/1413687641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e806d020 0x7fd3e80d0140 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0097a0 tx=0x7fd3dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed42ff81de212716 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:09.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3e5ffb640 1 -- 172.21.15.6:0/1413687641 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc035020 con 0x7fd3e806d020 2024-03-18T14:43:09.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3e5ffb640 1 -- 172.21.15.6:0/1413687641 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3dc031830 con 0x7fd3e806d020 2024-03-18T14:43:09.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3e5ffb640 1 -- 172.21.15.6:0/1413687641 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc031c00 con 0x7fd3e806d020 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/1413687641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e806d020 msgr2=0x7fd3e80d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/1413687641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e806d020 0x7fd3e80d0140 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0097a0 tx=0x7fd3dc02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/1413687641 shutdown_connections 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/1413687641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e806d020 0x7fd3e80d0140 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/1413687641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e80ce930 0x7fd3e806cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/1413687641 >> 172.21.15.6:0/1413687641 conn(0x7fd3e808b0c0 msgr2=0x7fd3e80d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/1413687641 shutdown_connections 2024-03-18T14:43:09.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.643+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/1413687641 wait complete. 2024-03-18T14:43:09.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.644+0000 7fd3ed5b3640 1 Processor -- start 2024-03-18T14:43:09.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.644+0000 7fd3ed5b3640 1 -- start start 2024-03-18T14:43:09.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3ed5b3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e806d020 0x7fd3e81f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3ed5b3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e80ce930 0x7fd3e814ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3ed5b3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3e80d0680 0x7fd3e80c5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3ed5b3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e80cee90 con 0x7fd3e80ce930 2024-03-18T14:43:09.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3ed5b3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e80ced10 con 0x7fd3e80d0680 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3ed5b3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e815ddc0 con 0x7fd3e806d020 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3e67fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e80ce930 0x7fd3e814ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3e67fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e80ce930 0x7fd3e814ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54042/0 (socket says 172.21.15.6:54042) 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.645+0000 7fd3e67fc640 1 -- 172.21.15.6:0/2715259680 learned_addr learned my addr 172.21.15.6:0/2715259680 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3e6ffd640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e806d020 0x7fd3e81f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3e67fc640 1 -- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e806d020 msgr2=0x7fd3e81f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3e67fc640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e806d020 0x7fd3e81f7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3e67fc640 1 -- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3e80d0680 msgr2=0x7fd3e80c5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3e67fc640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3e80d0680 0x7fd3e80c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3e67fc640 1 -- 172.21.15.6:0/2715259680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e81f8460 con 0x7fd3e80ce930 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3e67fc640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e80ce930 0x7fd3e814ed80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc02bee0 tx=0x7fd3dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3cf7fe640 1 -- 172.21.15.6:0/2715259680 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc03b980 con 0x7fd3e80ce930 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3cf7fe640 1 -- 172.21.15.6:0/2715259680 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3dc03bb20 con 0x7fd3e80ce930 2024-03-18T14:43:09.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e80b6900 con 0x7fd3e80ce930 2024-03-18T14:43:09.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3cf7fe640 1 -- 172.21.15.6:0/2715259680 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0453f0 con 0x7fd3e80ce930 2024-03-18T14:43:09.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.646+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e80b6ec0 con 0x7fd3e80ce930 2024-03-18T14:43:09.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.647+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e80751b0 con 0x7fd3e80ce930 2024-03-18T14:43:09.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.647+0000 7fd3cf7fe640 1 -- 172.21.15.6:0/2715259680 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd3dc038020 con 0x7fd3e80ce930 2024-03-18T14:43:09.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.648+0000 7fd3cf7fe640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3d406bbd0 0x7fd3d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:09.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.648+0000 7fd3cf7fe640 1 -- 172.21.15.6:0/2715259680 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fd3dc0ac8f0 con 0x7fd3e80ce930 2024-03-18T14:43:09.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.648+0000 7fd3e6ffd640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3d406bbd0 0x7fd3d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:09.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.648+0000 7fd3e6ffd640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3d406bbd0 0x7fd3d406e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009a00 tx=0x7fd3d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:09.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.652+0000 7fd3cf7fe640 1 -- 172.21.15.6:0/2715259680 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd3e81f8460 con 0x7fd3e80ce930 2024-03-18T14:43:09.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.866+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd3e808b680 con 0x7fd3e80ce930 2024-03-18T14:43:09.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.867+0000 7fd3cf7fe640 1 -- 172.21.15.6:0/2715259680 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fd3dc040070 con 0x7fd3e80ce930 2024-03-18T14:43:09.868 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3d406bbd0 msgr2=0x7fd3d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3d406bbd0 0x7fd3d406e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009a00 tx=0x7fd3d8008040 comp rx=0 tx=0).stop 2024-03-18T14:43:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e80ce930 msgr2=0x7fd3e814ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e80ce930 0x7fd3e814ed80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc02bee0 tx=0x7fd3dc0046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 shutdown_connections 2024-03-18T14:43:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd3d406bbd0 0x7fd3d406e090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3e80d0680 0x7fd3e80c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3e80ce930 0x7fd3e814ed80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 --2- 172.21.15.6:0/2715259680 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3e806d020 0x7fd3e81f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 >> 172.21.15.6:0/2715259680 conn(0x7fd3e808b0c0 msgr2=0x7fd3e80ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 shutdown_connections 2024-03-18T14:43:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.869+0000 7fd3ed5b3640 1 -- 172.21.15.6:0/2715259680 wait complete. 2024-03-18T14:43:09.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.984+0000 7fbbbde14640 1 Processor -- start 2024-03-18T14:43:10.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.984+0000 7fbbbde14640 1 -- start start 2024-03-18T14:43:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.985+0000 7fbbbde14640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbbb80d0730 0x7fbbb80cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.985+0000 7fbbbde14640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d2510 0x7fbbb815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.985+0000 7fbbbde14640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb80d2940 con 0x7fbbb80ce9d0 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.985+0000 7fbbbde14640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb80cd8d0 con 0x7fbbb80d0730 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.985+0000 7fbbbde14640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb80cfac0 con 0x7fbbb80d2510 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb7fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d2510 0x7fbbb815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb6ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbbb80d0730 0x7fbbb80cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb7fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d2510 0x7fbbb815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:37504/0 (socket says 172.21.15.6:37504) 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3262099963 learned_addr learned my addr 172.21.15.6:0/3262099963 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3262099963 >> v2:172.21.15.92:3300/0 conn(0x7fbbb80d0730 msgr2=0x7fbbb80cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb7fff640 1 --2- 172.21.15.6:0/3262099963 >> v2:172.21.15.92:3300/0 conn(0x7fbbb80d0730 0x7fbbb80cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3262099963 >> v1:172.21.15.92:6789/0 conn(0x7fbbb80ce9d0 legacy=0x7fbbb80d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.986+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3262099963 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb80cccc0 con 0x7fbbb80d2510 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.987+0000 7fbbb7fff640 1 --2- 172.21.15.6:0/3262099963 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d2510 0x7fbbb815b690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009e50 tx=0x7fbbb002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19dcbb66edb43cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.987+0000 7fbbb67fc640 1 -- 172.21.15.6:0/3262099963 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb002ca70 con 0x7fbbb80d2510 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.987+0000 7fbbb67fc640 1 -- 172.21.15.6:0/3262099963 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb002cc10 con 0x7fbbb80d2510 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.987+0000 7fbbb67fc640 1 -- 172.21.15.6:0/3262099963 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb00343f0 con 0x7fbbb80d2510 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 -- 172.21.15.6:0/3262099963 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d2510 msgr2=0x7fbbb815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3262099963 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d2510 0x7fbbb815b690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009e50 tx=0x7fbbb002bed0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 -- 172.21.15.6:0/3262099963 shutdown_connections 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3262099963 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d2510 0x7fbbb815b690 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3262099963 >> v2:172.21.15.92:3300/0 conn(0x7fbbb80d0730 0x7fbbb80cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 -- 172.21.15.6:0/3262099963 >> 172.21.15.6:0/3262099963 conn(0x7fbbb808b0c0 msgr2=0x7fbbb80d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 -- 172.21.15.6:0/3262099963 shutdown_connections 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.988+0000 7fbbbde14640 1 -- 172.21.15.6:0/3262099963 wait complete. 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.989+0000 7fbbbde14640 1 Processor -- start 2024-03-18T14:43:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.989+0000 7fbbbde14640 1 -- start start 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbbde14640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80ce9d0 0x7fbbb80b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbbde14640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d0730 0x7fbbb80b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbbde14640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbb80d2510 0x7fbbb80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbbde14640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80cd560 con 0x7fbbb80d2510 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbbde14640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80cd280 con 0x7fbbb80ce9d0 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbbde14640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb80cedb0 con 0x7fbbb80d0730 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbb80d2510 0x7fbbb80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbb80d2510 0x7fbbb80cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54054/0 (socket says 172.21.15.6:54054) 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3131763174 learned_addr learned my addr 172.21.15.6:0/3131763174 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb6ffd640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d0730 0x7fbbb80b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb77fe640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80ce9d0 0x7fbbb80b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d0730 msgr2=0x7fbbb80b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d0730 0x7fbbb80b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80ce9d0 msgr2=0x7fbbb80b7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80ce9d0 0x7fbbb80b7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 -- 172.21.15.6:0/3131763174 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb80b4b40 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbb7fff640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbb80d2510 0x7fbbb80cae00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009f70 tx=0x7fbbb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbb9ffff640 1 -- 172.21.15.6:0/3131763174 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0043020 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbb9ffff640 1 -- 172.21.15.6:0/3131763174 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb003e450 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.990+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb80ca040 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.991+0000 7fbb9ffff640 1 -- 172.21.15.6:0/3131763174 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0009510 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.991+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb80ca600 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.992+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb80751b0 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.992+0000 7fbb9ffff640 1 -- 172.21.15.6:0/3131763174 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbbb0046020 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.992+0000 7fbb9ffff640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbba406bbd0 0x7fbba406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.992+0000 7fbb9ffff640 1 -- 172.21.15.6:0/3131763174 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fbbb00ad3b0 con 0x7fbbb80d2510 2024-03-18T14:43:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.993+0000 7fbbb77fe640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbba406bbd0 0x7fbba406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.993+0000 7fbbb77fe640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbba406bbd0 0x7fbba406e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbba8007fd0 tx=0x7fbba8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:09.997+0000 7fbb9ffff640 1 -- 172.21.15.6:0/3131763174 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbbb004d030 con 0x7fbbb80d2510 2024-03-18T14:43:10.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.181+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbbb80cfe00 con 0x7fbbb80d2510 2024-03-18T14:43:10.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.181+0000 7fbb9ffff640 1 -- 172.21.15.6:0/3131763174 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fbbb007b540 con 0x7fbbb80d2510 2024-03-18T14:43:10.182 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbba406bbd0 msgr2=0x7fbba406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbba406bbd0 0x7fbba406e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbba8007fd0 tx=0x7fbba8009040 comp rx=0 tx=0).stop 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbb80d2510 msgr2=0x7fbbb80cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbb80d2510 0x7fbbb80cae00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009f70 tx=0x7fbbb0004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 shutdown_connections 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbba406bbd0 0x7fbba406e090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbb80d2510 0x7fbbb80cae00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80d0730 0x7fbbb80b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 --2- 172.21.15.6:0/3131763174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80ce9d0 0x7fbbb80b7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 >> 172.21.15.6:0/3131763174 conn(0x7fbbb808b0c0 msgr2=0x7fbbb80ae610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 shutdown_connections 2024-03-18T14:43:10.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.183+0000 7fbbbde14640 1 -- 172.21.15.6:0/3131763174 wait complete. 2024-03-18T14:43:10.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-18T14:43:10.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.336+0000 7fda1cdc4640 1 Processor -- start 2024-03-18T14:43:10.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.336+0000 7fda1cdc4640 1 -- start start 2024-03-18T14:43:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda1cdc4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda1815bdd0 0x7fda180d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda1cdc4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fda180cd8f0 0x7fda180b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda1cdc4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda180cdf60 con 0x7fda180d05a0 2024-03-18T14:43:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda1cdc4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fda180d0a00 con 0x7fda1815bdd0 2024-03-18T14:43:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda1cdc4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fda180ce240 con 0x7fda180cd8f0 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda1659c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda1815bdd0 0x7fda180d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda15d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fda180d05a0 0x7fda180cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37536/0 (socket says 172.21.15.6:37536) 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.337+0000 7fda15d9b640 1 -- 172.21.15.6:0/1660154630 learned_addr learned my addr 172.21.15.6:0/1660154630 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.338+0000 7fda16d9d640 1 --2- 172.21.15.6:0/1660154630 >> v2:172.21.15.92:3300/0 conn(0x7fda180cd8f0 0x7fda180b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.338+0000 7fda1659c640 1 -- 172.21.15.6:0/1660154630 >> v2:172.21.15.92:3300/0 conn(0x7fda180cd8f0 msgr2=0x7fda180b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.338+0000 7fda1659c640 1 --2- 172.21.15.6:0/1660154630 >> v2:172.21.15.92:3300/0 conn(0x7fda180cd8f0 0x7fda180b2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.338+0000 7fda1659c640 1 -- 172.21.15.6:0/1660154630 >> v1:172.21.15.6:6789/0 conn(0x7fda180d05a0 legacy=0x7fda180cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.338+0000 7fda1659c640 1 -- 172.21.15.6:0/1660154630 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda180b1780 con 0x7fda1815bdd0 2024-03-18T14:43:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.338+0000 7fda1659c640 1 --2- 172.21.15.6:0/1660154630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1815bdd0 0x7fda180d0060 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fda000097a0 tx=0x7fda0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5087a1d4af37dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.338+0000 7fda1559a640 1 -- 172.21.15.6:0/1660154630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00035020 con 0x7fda1815bdd0 2024-03-18T14:43:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1559a640 1 -- 172.21.15.6:0/1660154630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda00031830 con 0x7fda1815bdd0 2024-03-18T14:43:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1559a640 1 -- 172.21.15.6:0/1660154630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00031c00 con 0x7fda1815bdd0 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1660154630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1815bdd0 msgr2=0x7fda180d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1660154630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1815bdd0 0x7fda180d0060 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fda000097a0 tx=0x7fda0002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1660154630 shutdown_connections 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1660154630 >> v2:172.21.15.92:3300/0 conn(0x7fda180cd8f0 0x7fda180b2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1660154630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda1815bdd0 0x7fda180d0060 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1660154630 >> 172.21.15.6:0/1660154630 conn(0x7fda1809ae00 msgr2=0x7fda1808b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1660154630 shutdown_connections 2024-03-18T14:43:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.339+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1660154630 wait complete. 2024-03-18T14:43:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 Processor -- start 2024-03-18T14:43:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 -- start start 2024-03-18T14:43:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda180cd8f0 0x7fda180c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda180d05a0 0x7fda1806c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1815bdd0 0x7fda180c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda180b27a0 con 0x7fda180cd8f0 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda1815c1b0 con 0x7fda1815bdd0 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.341+0000 7fda1cdc4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fda180b2920 con 0x7fda180d05a0 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda180cd8f0 0x7fda180c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda180cd8f0 0x7fda180c7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54074/0 (socket says 172.21.15.6:54074) 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 -- 172.21.15.6:0/1681624938 learned_addr learned my addr 172.21.15.6:0/1681624938 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda16d9d640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1815bdd0 0x7fda180c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda15d9b640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda180d05a0 0x7fda1806c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 -- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda180d05a0 msgr2=0x7fda1806c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda180d05a0 0x7fda1806c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 -- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1815bdd0 msgr2=0x7fda180c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1815bdd0 0x7fda180c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 -- 172.21.15.6:0/1681624938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda180c8530 con 0x7fda180cd8f0 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda16d9d640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1815bdd0 0x7fda180c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda15d9b640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda180d05a0 0x7fda1806c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1659c640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda180cd8f0 0x7fda180c7ff0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fda00034040 tx=0x7fda000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda0effd640 1 -- 172.21.15.6:0/1681624938 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00040070 con 0x7fda180cd8f0 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda0effd640 1 -- 172.21.15.6:0/1681624938 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0003e070 con 0x7fda180cd8f0 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1806cce0 con 0x7fda180cd8f0 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda0effd640 1 -- 172.21.15.6:0/1681624938 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00035040 con 0x7fda180cd8f0 2024-03-18T14:43:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.342+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1806d2a0 con 0x7fda180cd8f0 2024-03-18T14:43:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.343+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda180751b0 con 0x7fda180cd8f0 2024-03-18T14:43:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.344+0000 7fda0effd640 1 -- 172.21.15.6:0/1681624938 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fda00038020 con 0x7fda180cd8f0 2024-03-18T14:43:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.344+0000 7fda0effd640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9f806bc00 0x7fd9f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.344+0000 7fda0effd640 1 -- 172.21.15.6:0/1681624938 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fda000ac6d0 con 0x7fda180cd8f0 2024-03-18T14:43:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.345+0000 7fda15d9b640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9f806bc00 0x7fd9f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.345+0000 7fda15d9b640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9f806bc00 0x7fd9f806e0c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fda180cf120 tx=0x7fda08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.349+0000 7fda0effd640 1 -- 172.21.15.6:0/1681624938 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fda0007aec0 con 0x7fda180cd8f0 2024-03-18T14:43:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.651+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fda18070980 con 0x7fda180cd8f0 2024-03-18T14:43:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.651+0000 7fda0effd640 1 -- 172.21.15.6:0/1681624938 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) ==== 97+0+11 (secure 0 0 0) 0x7fda0007a7c0 con 0x7fda180cd8f0 2024-03-18T14:43:10.652 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9f806bc00 msgr2=0x7fd9f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9f806bc00 0x7fd9f806e0c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fda180cf120 tx=0x7fda08009040 comp rx=0 tx=0).stop 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda180cd8f0 msgr2=0x7fda180c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda180cd8f0 0x7fda180c7ff0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fda00034040 tx=0x7fda000046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 shutdown_connections 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd9f806bc00 0x7fd9f806e0c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1815bdd0 0x7fda180c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda180d05a0 0x7fda1806c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 --2- 172.21.15.6:0/1681624938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda180cd8f0 0x7fda180c7ff0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 >> 172.21.15.6:0/1681624938 conn(0x7fda1809ae00 msgr2=0x7fda1815b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 shutdown_connections 2024-03-18T14:43:10.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.653+0000 7fda1cdc4640 1 -- 172.21.15.6:0/1681624938 wait complete. 2024-03-18T14:43:10.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:10.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.766+0000 7f50705f4640 1 Processor -- start 2024-03-18T14:43:10.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.766+0000 7f50705f4640 1 -- start start 2024-03-18T14:43:10.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.767+0000 7f50705f4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f50680d2180 0x7f50680cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.767+0000 7f50705f4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50680cd450 con 0x7f506815f2a0 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.767+0000 7f50705f4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50680cd730 con 0x7f50680ccc80 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.767+0000 7f50705f4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f506815f680 con 0x7f50680d2180 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.767+0000 7f506e390640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f506815f2a0 0x7f50680d1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37546/0 (socket says 172.21.15.6:37546) 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.767+0000 7f506e390640 1 -- 172.21.15.6:0/59876373 learned_addr learned my addr 172.21.15.6:0/59876373 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506db8f640 1 --2- 172.21.15.6:0/59876373 >> v2:172.21.15.92:3300/0 conn(0x7f50680d2180 0x7f50680cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3926178373 0 0) 0x7f50680cd450 con 0x7f506815f2a0 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5054003520 con 0x7f506815f2a0 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3520728688 0 0) 0x7f50680cd730 con 0x7f50680ccc80 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50680cd450 con 0x7f50680ccc80 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2496043851 0 0) 0x7f5054003520 con 0x7f506815f2a0 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506db8f640 1 -- 172.21.15.6:0/59876373 >> v1:172.21.15.92:6789/0 conn(0x7f50680ccc80 legacy=0x7f50680c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506db8f640 1 -- 172.21.15.6:0/59876373 >> v1:172.21.15.6:6789/0 conn(0x7f506815f2a0 legacy=0x7f50680d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.768+0000 7f506db8f640 1 -- 172.21.15.6:0/59876373 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50680cda10 con 0x7f50680d2180 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.769+0000 7f506db8f640 1 --2- 172.21.15.6:0/59876373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f50680cc740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f506815f800 tx=0x7f505802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c59ca91649d27d8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.769+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058039070 con 0x7f50680d2180 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.769+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50580314f0 con 0x7f50680d2180 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.769+0000 7f506d38e640 1 -- 172.21.15.6:0/59876373 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064001670 con 0x7f50680d2180 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.770+0000 7f50705f4640 1 -- 172.21.15.6:0/59876373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 msgr2=0x7f50680cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.770+0000 7f50705f4640 1 --2- 172.21.15.6:0/59876373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f50680cc740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f506815f800 tx=0x7f505802b810 comp rx=0 tx=0).stop 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.770+0000 7f50705f4640 1 -- 172.21.15.6:0/59876373 shutdown_connections 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.770+0000 7f50705f4640 1 --2- 172.21.15.6:0/59876373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f50680cc740 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.770+0000 7f50705f4640 1 -- 172.21.15.6:0/59876373 >> 172.21.15.6:0/59876373 conn(0x7f506809ae00 msgr2=0x7f506808b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:10.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.770+0000 7f50705f4640 1 -- 172.21.15.6:0/59876373 shutdown_connections 2024-03-18T14:43:10.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.770+0000 7f50705f4640 1 -- 172.21.15.6:0/59876373 wait complete. 2024-03-18T14:43:10.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.771+0000 7f50705f4640 1 Processor -- start 2024-03-18T14:43:10.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.771+0000 7f50705f4640 1 -- start start 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50680ccc80 0x7f50681f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f506815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f506815f2a0 0x7f506806c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f50680cd0f0 con 0x7f50680ccc80 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f50680cdfd0 con 0x7f50680d2180 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f50680c6000 con 0x7f506815f2a0 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50680ccc80 0x7f50681f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50680ccc80 0x7f50681f7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54078/0 (socket says 172.21.15.6:54078) 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 -- 172.21.15.6:0/1291334210 learned_addr learned my addr 172.21.15.6:0/1291334210 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506eb91640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f506815f2a0 0x7f506806c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506db8f640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f506815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 -- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f506815f2a0 msgr2=0x7f506806c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f506815f2a0 0x7f506806c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 -- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 msgr2=0x7f506815b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f506815b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 -- 172.21.15.6:0/1291334210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50681f39b0 con 0x7f50680ccc80 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506eb91640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f506815f2a0 0x7f506806c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506db8f640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f506815b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f506e390640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50680ccc80 0x7f50681f7360 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f506400c520 tx=0x7f506400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f505effd640 1 -- 172.21.15.6:0/1291334210 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064017070 con 0x7f50680ccc80 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f505effd640 1 -- 172.21.15.6:0/1291334210 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5064005d00 con 0x7f50680ccc80 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50681f3c90 con 0x7f50680ccc80 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f505effd640 1 -- 172.21.15.6:0/1291334210 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50640072b0 con 0x7f50680ccc80 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.772+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50681f4250 con 0x7f50680ccc80 2024-03-18T14:43:10.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.773+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f506804f5b0 con 0x7f50680ccc80 2024-03-18T14:43:10.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.774+0000 7f505effd640 1 -- 172.21.15.6:0/1291334210 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f506401a070 con 0x7f50680ccc80 2024-03-18T14:43:10.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.774+0000 7f505effd640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f504c06bbd0 0x7f504c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:10.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.775+0000 7f505effd640 1 -- 172.21.15.6:0/1291334210 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f506408a2d0 con 0x7f50680ccc80 2024-03-18T14:43:10.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.775+0000 7f506db8f640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f504c06bbd0 0x7f504c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:10.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.775+0000 7f506db8f640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f504c06bbd0 0x7f504c06e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:10.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:10.779+0000 7f505effd640 1 -- 172.21.15.6:0/1291334210 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5064058ab0 con 0x7f50680ccc80 2024-03-18T14:43:11.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.090+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f506806eb10 con 0x7f50680ccc80 2024-03-18T14:43:11.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.091+0000 7f505effd640 1 -- 172.21.15.6:0/1291334210 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) ==== 80+0+0 (secure 0 0 0) 0x7f50640583b0 con 0x7f50680ccc80 2024-03-18T14:43:11.092 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f504c06bbd0 msgr2=0x7f504c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f504c06bbd0 0x7f504c06e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058008030 comp rx=0 tx=0).stop 2024-03-18T14:43:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50680ccc80 msgr2=0x7f50681f7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50680ccc80 0x7f50681f7360 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f506400c520 tx=0x7f506400c9e0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 shutdown_connections 2024-03-18T14:43:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f504c06bbd0 0x7f504c06e090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f506815f2a0 0x7f506806c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50680d2180 0x7f506815b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 --2- 172.21.15.6:0/1291334210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50680ccc80 0x7f50681f7360 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 >> 172.21.15.6:0/1291334210 conn(0x7f506809ae00 msgr2=0x7f506815d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:11.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 shutdown_connections 2024-03-18T14:43:11.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.093+0000 7f50705f4640 1 -- 172.21.15.6:0/1291334210 wait complete. 2024-03-18T14:43:11.105 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:43:11.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:11.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:11.131 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[54476]: fuse finished with error 0 and tester_r 0 2024-03-18T14:43:11.133 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:11.225 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:43:11.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:11.226 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:43:11.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:43:11.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.385+0000 7f4afbcf8640 1 Processor -- start 2024-03-18T14:43:11.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.385+0000 7f4afbcf8640 1 -- start start 2024-03-18T14:43:11.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.385+0000 7f4afbcf8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4af40d1f60 0x7f4af40cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.385+0000 7f4afbcf8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 0x7f4af40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.385+0000 7f4afbcf8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af40cc870 con 0x7f4af415bd40 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.385+0000 7f4afbcf8640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f4af40d0500 con 0x7f4af40d1f60 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.385+0000 7f4afbcf8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4af40d07e0 con 0x7f4af40cfcd0 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af9a94640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4af415bd40 0x7f4af40d1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37548/0 (socket says 172.21.15.6:37548) 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af9a94640 1 -- 172.21.15.6:0/710262978 learned_addr learned my addr 172.21.15.6:0/710262978 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430310912 0 0) 0x7f4af40cc870 con 0x7f4af415bd40 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae0003520 con 0x7f4af415bd40 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af9293640 1 --2- 172.21.15.6:0/710262978 >> v2:172.21.15.92:3300/0 conn(0x7f4af40d1f60 0x7f4af40cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4afa295640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 0x7f4af40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3804817849 0 0) 0x7f4ae0003520 con 0x7f4af415bd40 2024-03-18T14:43:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 msgr2=0x7f4af40cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 0x7f4af40cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 >> v2:172.21.15.92:3300/0 conn(0x7f4af40d1f60 msgr2=0x7f4af40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 --2- 172.21.15.6:0/710262978 >> v2:172.21.15.92:3300/0 conn(0x7f4af40d1f60 0x7f4af40cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af40ccb10 con 0x7f4af415bd40 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f4af0001670 con 0x7f4af415bd40 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 >> v1:172.21.15.6:6789/0 conn(0x7f4af415bd40 legacy=0x7f4af40d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 0x7f4ae00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ae0003f80 0x7f4ae0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ae0004d40 0x7f4ae00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4af40d0500 con 0x7f4ae0004d40 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4af40d07e0 con 0x7f4ae0003f80 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae000ba70 con 0x7f4af40cfcd0 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4afa295640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ae0004d40 0x7f4ae00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af9a94640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 0x7f4ae00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.386+0000 7f4af9293640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ae0003f80 0x7f4ae0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.387+0000 7f4afa295640 1 -- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 msgr2=0x7f4ae00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.387+0000 7f4afa295640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af40cfcd0 0x7f4ae00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.387+0000 7f4afa295640 1 -- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ae0003f80 msgr2=0x7f4ae0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.387+0000 7f4afa295640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ae0003f80 0x7f4ae0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.387+0000 7f4afa295640 1 -- 172.21.15.6:0/710262978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae0003520 con 0x7f4ae0004d40 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.387+0000 7f4afa295640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ae0004d40 0x7f4ae00051c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009fc0 tx=0x7f4ae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd8c239220f357 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.387+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec039070 con 0x7f4ae0004d40 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aec02cc20 con 0x7f4ae0004d40 2024-03-18T14:43:11.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4af8a92640 1 -- 172.21.15.6:0/710262978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec009c10 con 0x7f4ae0004d40 2024-03-18T14:43:11.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4afbcf8640 1 -- 172.21.15.6:0/710262978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ae0004d40 msgr2=0x7f4ae00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4afbcf8640 1 --2- 172.21.15.6:0/710262978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ae0004d40 0x7f4ae00051c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4aec009fc0 tx=0x7f4ae0005830 comp rx=0 tx=0).stop 2024-03-18T14:43:11.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4afa295640 1 -- 172.21.15.6:0/710262978 reap_dead start 2024-03-18T14:43:11.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4afbcf8640 1 -- 172.21.15.6:0/710262978 shutdown_connections 2024-03-18T14:43:11.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4afbcf8640 1 -- 172.21.15.6:0/710262978 >> 172.21.15.6:0/710262978 conn(0x7f4af409ae20 msgr2=0x7f4af408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:11.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4afbcf8640 1 -- 172.21.15.6:0/710262978 shutdown_connections 2024-03-18T14:43:11.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.388+0000 7f4afbcf8640 1 -- 172.21.15.6:0/710262978 wait complete. 2024-03-18T14:43:11.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.389+0000 7f4afbcf8640 1 Processor -- start 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.389+0000 7f4afbcf8640 1 -- start start 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4afbcf8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af4151390 0x7f4af40af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4afbcf8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af40cb5c0 0x7f4af415e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4afbcf8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af415eeb0 0x7f4af40c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4afbcf8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4af40d02c0 con 0x7f4af40cb5c0 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4afbcf8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4af40d0140 con 0x7f4af415eeb0 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4afbcf8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4af415c120 con 0x7f4af4151390 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4af9293640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af40cb5c0 0x7f4af415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4af9293640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af40cb5c0 0x7f4af415e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54090/0 (socket says 172.21.15.6:54090) 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4af9293640 1 -- 172.21.15.6:0/3295197485 learned_addr learned my addr 172.21.15.6:0/3295197485 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4af9a94640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af4151390 0x7f4af40af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.390+0000 7f4afa295640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af415eeb0 0x7f4af40c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4af9293640 1 -- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af4151390 msgr2=0x7f4af40af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4af9293640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af4151390 0x7f4af40af760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4af9293640 1 -- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af415eeb0 msgr2=0x7f4af40c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4af9293640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af415eeb0 0x7f4af40c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4af9293640 1 -- 172.21.15.6:0/3295197485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af415dbf0 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4af9293640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af40cb5c0 0x7f4af415e970 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4011a70 tx=0x7f4ae4011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4aea7fc640 1 -- 172.21.15.6:0/3295197485 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae400fa90 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4aea7fc640 1 -- 172.21.15.6:0/3295197485 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae400fc30 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af415de90 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4aea7fc640 1 -- 172.21.15.6:0/3295197485 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4014440 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.391+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af40cbaf0 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.392+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af415e060 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.392+0000 7f4aea7fc640 1 -- 172.21.15.6:0/3295197485 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4ae4012040 con 0x7f4af40cb5c0 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.393+0000 7f4aea7fc640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ad806bbd0 0x7f4ad806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:11.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.393+0000 7f4aea7fc640 1 -- 172.21.15.6:0/3295197485 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f4ae408b390 con 0x7f4af40cb5c0 2024-03-18T14:43:11.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.393+0000 7f4af9a94640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ad806bbd0 0x7f4ad806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:11.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.394+0000 7f4af9a94640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ad806bbd0 0x7f4ad806e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4af0004840 tx=0x7f4af0001e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:11.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.397+0000 7f4aea7fc640 1 -- 172.21.15.6:0/3295197485 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4ae4019030 con 0x7f4af40cb5c0 2024-03-18T14:43:11.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.621+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4af40b7a10 con 0x7f4af40cb5c0 2024-03-18T14:43:11.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.621+0000 7f4aea7fc640 1 -- 172.21.15.6:0/3295197485 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+82 (secure 0 0 0) 0x7f4ae4059b70 con 0x7f4af40cb5c0 2024-03-18T14:43:11.623 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ad806bbd0 msgr2=0x7f4ad806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ad806bbd0 0x7f4ad806e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4af0004840 tx=0x7f4af0001e40 comp rx=0 tx=0).stop 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af40cb5c0 msgr2=0x7f4af415e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af40cb5c0 0x7f4af415e970 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4011a70 tx=0x7f4ae4011e70 comp rx=0 tx=0).stop 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 shutdown_connections 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f4ad806bbd0 0x7f4ad806e090 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4af415eeb0 0x7f4af40c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4af40cb5c0 0x7f4af415e970 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 --2- 172.21.15.6:0/3295197485 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4af4151390 0x7f4af40af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 >> 172.21.15.6:0/3295197485 conn(0x7f4af409ae20 msgr2=0x7f4af40d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 shutdown_connections 2024-03-18T14:43:11.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11.623+0000 7f4afbcf8640 1 -- 172.21.15.6:0/3295197485 wait complete. 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:43:11.635 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:43:11.636 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:11.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:11.636 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:11.651 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42574sec preferred_lft 42574sec 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:43:11.652 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c41d:adff:fe29:e32/64 scope link 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::d035:11ff:fe58:6e4e/64 scope link 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::9023:d7ff:fecf:2a25/64 scope link 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:11.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:11.653 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:43:11.716 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:43:11.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:11.718 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:43:11.773 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:43:11.773 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:43:11.773 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:43:11.773 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:43:11.773 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:43:11.773 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:43:11.774 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:43:11.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:43:11.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:43:11.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:43:11.774 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:43:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:43:11.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:11.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:11.909 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:43:11.923 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T14:43:11.924 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:43:11.924 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:11.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:11.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:43:11.991 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T14:43:11.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:11.991 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:12.046 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:12.046 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:12.046 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:12.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:12.170 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:12.226 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:12.227 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:12.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:12.228 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:12.280 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:43:12.280 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T14:43:12.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:12.353 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:12.409 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:12.409 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:12.410 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:12.411 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:12.412 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:12.412 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:12.412 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:12.412 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:12.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:12.412 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:12.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:2024-03-18T14:43:12.411+0000 7f1683037480 -1 init, newargv = 0x558207957ec0 newargc=15 2024-03-18T14:43:12.413 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[55729]: starting ceph client 2024-03-18T14:43:12.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[55729]: starting fuse 2024-03-18T14:43:12.427 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:43:12.428 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:43:12.428 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.0")) 2024-03-18T14:43:12.429 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:43:12.559 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.0.55729.asok 2024-03-18T14:43:12.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:12.569 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55729.asok 2024-03-18T14:43:12.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:12.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55729.asok status 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "0", 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "55729", 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5877, 2024-03-18T14:43:12.707 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5877 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3287493471 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3287493471 2024-03-18T14:43:12.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.5877 192.168.0.1:0/3287493471", 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/3287493471", 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 16, 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 42, 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:43:12.709 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:12.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:12.715 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:43:12.715 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:12.729 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:43:12.730 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:12.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:12.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:12.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:12.797 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-18T14:43:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:43:12.853 INFO:teuthology.orchestra.run.smithi006.stderr:setfattr: layoutfile: Permission denied 2024-03-18T14:43:12.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:12.853 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-18T14:43:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:43:12.909 INFO:teuthology.orchestra.run.smithi006.stderr:setfattr: layoutdir: Permission denied 2024-03-18T14:43:12.909 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:12.909 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:12.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.103+0000 7fb833c0c640 1 Processor -- start 2024-03-18T14:43:13.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.103+0000 7fb833c0c640 1 -- start start 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.104+0000 7fb833c0c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb82c0d1f60 0x7fb82c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.104+0000 7fb833c0c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c0cc870 con 0x7fb82c15bd40 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.104+0000 7fb833c0c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c0d0500 con 0x7fb82c0cfcd0 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.104+0000 7fb833c0c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb82c0d07e0 con 0x7fb82c0d1f60 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8319a8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb82c15bd40 0x7fb82c0d1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37552/0 (socket says 172.21.15.6:37552) 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8319a8640 1 -- 172.21.15.6:0/1499847486 learned_addr learned my addr 172.21.15.6:0/1499847486 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4084755797 0 0) 0x7fb82c0cc870 con 0x7fb82c15bd40 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8311a7640 1 --2- 172.21.15.6:0/1499847486 >> v2:172.21.15.92:3300/0 conn(0x7fb82c0d1f60 0x7fb82c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb818003520 con 0x7fb82c15bd40 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3589906420 0 0) 0x7fb818003520 con 0x7fb82c15bd40 2024-03-18T14:43:13.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 >> v2:172.21.15.92:3300/0 conn(0x7fb82c0d1f60 msgr2=0x7fb82c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 --2- 172.21.15.6:0/1499847486 >> v2:172.21.15.92:3300/0 conn(0x7fb82c0d1f60 0x7fb82c0cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 >> v1:172.21.15.92:6789/0 conn(0x7fb82c0cfcd0 legacy=0x7fb82c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c0ccb10 con 0x7fb82c15bd40 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fb828001670 con 0x7fb82c15bd40 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 >> v1:172.21.15.6:6789/0 conn(0x7fb82c15bd40 legacy=0x7fb82c0d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8180039b0 0x7fb818003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb818004800 0x7fb818004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb81800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c0d07e0 con 0x7fb8180051c0 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb81800c160 con 0x7fb818004800 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb81800c360 con 0x7fb8180039b0 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8321a9640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb81800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.105+0000 7fb8319a8640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8180039b0 0x7fb818003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8311a7640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb818004800 0x7fb818004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8321a9640 1 -- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8180039b0 msgr2=0x7fb818003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8321a9640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8180039b0 0x7fb818003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8321a9640 1 -- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb818004800 msgr2=0x7fb818004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8321a9640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb818004800 0x7fb818004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8321a9640 1 -- 172.21.15.6:0/1499847486 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb818003520 con 0x7fb8180051c0 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8321a9640 1 -- 172.21.15.6:0/1499847486 reap_dead start 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.106+0000 7fb8321a9640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb81800b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb824001e20 tx=0x7fb818005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6990dd3ba59753 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824001a70 con 0x7fb8180051c0 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb824003b70 con 0x7fb8180051c0 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb8309a6640 1 -- 172.21.15.6:0/1499847486 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8240050d0 con 0x7fb8180051c0 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb833c0c640 1 -- 172.21.15.6:0/1499847486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 msgr2=0x7fb81800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb81800b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb824001e20 tx=0x7fb818005830 comp rx=0 tx=0).stop 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb833c0c640 1 -- 172.21.15.6:0/1499847486 shutdown_connections 2024-03-18T14:43:13.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1499847486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb81800b9c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb833c0c640 1 -- 172.21.15.6:0/1499847486 >> 172.21.15.6:0/1499847486 conn(0x7fb82c09ae20 msgr2=0x7fb82c08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:13.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb833c0c640 1 -- 172.21.15.6:0/1499847486 shutdown_connections 2024-03-18T14:43:13.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.107+0000 7fb833c0c640 1 -- 172.21.15.6:0/1499847486 wait complete. 2024-03-18T14:43:13.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.108+0000 7fb833c0c640 1 Processor -- start 2024-03-18T14:43:13.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.108+0000 7fb833c0c640 1 -- start start 2024-03-18T14:43:13.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb833c0c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb82c14e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb833c0c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb82c0b09e0 0x7fb82c0c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb833c0c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb82c0b10c0 0x7fb82c0c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb833c0c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c0d0280 con 0x7fb8180051c0 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb833c0c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c0d0100 con 0x7fb82c0b09e0 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb833c0c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb82c15c120 con 0x7fb82c0b10c0 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb8319a8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb82c14e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb8319a8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb82c14e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54106/0 (socket says 172.21.15.6:54106) 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb8319a8640 1 -- 172.21.15.6:0/1364993489 learned_addr learned my addr 172.21.15.6:0/1364993489 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb8321a9640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb82c0b10c0 0x7fb82c0c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.109+0000 7fb8311a7640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb82c0b09e0 0x7fb82c0c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8319a8640 1 -- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb82c0b10c0 msgr2=0x7fb82c0c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8319a8640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb82c0b10c0 0x7fb82c0c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8319a8640 1 -- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb82c0b09e0 msgr2=0x7fb82c0c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8319a8640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb82c0b09e0 0x7fb82c0c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8319a8640 1 -- 172.21.15.6:0/1364993489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c0b7080 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8311a7640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb82c0b09e0 0x7fb82c0c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8319a8640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb82c14e0b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb8280073a0 tx=0x7fb828007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8227fc640 1 -- 172.21.15.6:0/1364993489 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82801a070 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8227fc640 1 -- 172.21.15.6:0/1364993489 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82800f960 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82c0b7360 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb8227fc640 1 -- 172.21.15.6:0/1364993489 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82800fd50 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.110+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb82c0b5c40 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.111+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb82c09fa90 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.111+0000 7fb8227fc640 1 -- 172.21.15.6:0/1364993489 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb82801e3f0 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.112+0000 7fb8227fc640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb81006bc00 0x7fb81006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.112+0000 7fb8227fc640 1 -- 172.21.15.6:0/1364993489 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fb82808a590 con 0x7fb8180051c0 2024-03-18T14:43:13.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.112+0000 7fb8311a7640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb81006bc00 0x7fb81006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.113+0000 7fb8311a7640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb81006bc00 0x7fb81006e0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0093d0 tx=0x7fb81c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.116+0000 7fb8227fc640 1 -- 172.21.15.6:0/1364993489 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb828058d70 con 0x7fb8180051c0 2024-03-18T14:43:13.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.365+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb82c04f620 con 0x7fb8180051c0 2024-03-18T14:43:13.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.366+0000 7fb8227fc640 1 -- 172.21.15.6:0/1364993489 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+18658 (secure 0 0 0) 0x7fb828058670 con 0x7fb8180051c0 2024-03-18T14:43:13.368 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:13.368 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:06.352665+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:13.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb81006bc00 msgr2=0x7fb81006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb81006bc00 0x7fb81006e0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0093d0 tx=0x7fb81c008040 comp rx=0 tx=0).stop 2024-03-18T14:43:13.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 msgr2=0x7fb82c14e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb82c14e0b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb8280073a0 tx=0x7fb828007860 comp rx=0 tx=0).stop 2024-03-18T14:43:13.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 shutdown_connections 2024-03-18T14:43:13.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb81006bc00 0x7fb81006e0c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb82c0b10c0 0x7fb82c0c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb82c0b09e0 0x7fb82c0c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 --2- 172.21.15.6:0/1364993489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8180051c0 0x7fb82c14e0b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 >> 172.21.15.6:0/1364993489 conn(0x7fb82c09ae20 msgr2=0x7fb82c15a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:13.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 shutdown_connections 2024-03-18T14:43:13.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.368+0000 7fb833c0c640 1 -- 172.21.15.6:0/1364993489 wait complete. 2024-03-18T14:43:13.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:42:54.939648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:42:55.638120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-18T14:43:03.985735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:13.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.484+0000 7fe6d1c19640 1 Processor -- start 2024-03-18T14:43:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6d1c19640 1 -- start start 2024-03-18T14:43:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6d1c19640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe6cc15e030 0x7fe6cc0d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6d1c19640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6cc0b6140 0x7fe6cc0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6d1c19640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6d1c19640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe6cc15e410 con 0x7fe6cc0b6140 2024-03-18T14:43:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6d1c19640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe6cc0ce300 con 0x7fe6cc15e030 2024-03-18T14:43:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6d1c19640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe6cc0d0e20 con 0x7fe6cc0d0450 2024-03-18T14:43:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.485+0000 7fe6c3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6cc0b6140 0x7fe6cc0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6c3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6cc0b6140 0x7fe6cc0ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54116/0 (socket says 172.21.15.6:54116) 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6c3fff640 1 -- 172.21.15.6:0/4241724198 learned_addr learned my addr 172.21.15.6:0/4241724198 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6cbfff640 1 --2- 172.21.15.6:0/4241724198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6cb7fe640 1 --2- 172.21.15.6:0/4241724198 >> v2:172.21.15.92:3300/0 conn(0x7fe6cc15e030 0x7fe6cc0d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6c3fff640 1 -- 172.21.15.6:0/4241724198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 msgr2=0x7fe6cc0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6c3fff640 1 --2- 172.21.15.6:0/4241724198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6c3fff640 1 -- 172.21.15.6:0/4241724198 >> v2:172.21.15.92:3300/0 conn(0x7fe6cc15e030 msgr2=0x7fe6cc0d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6c3fff640 1 --2- 172.21.15.6:0/4241724198 >> v2:172.21.15.92:3300/0 conn(0x7fe6cc15e030 0x7fe6cc0d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6c3fff640 1 -- 172.21.15.6:0/4241724198 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6cc0b65a0 con 0x7fe6cc0b6140 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.486+0000 7fe6cb7fe640 1 --2- 172.21.15.6:0/4241724198 >> v2:172.21.15.92:3300/0 conn(0x7fe6cc15e030 0x7fe6cc0d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6c3fff640 1 --2- 172.21.15.6:0/4241724198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc0ceec0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc068ad0 tx=0x7fe6b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2f950a4905d420 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6caffd640 1 -- 172.21.15.6:0/4241724198 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4035020 con 0x7fe6cc0b6140 2024-03-18T14:43:13.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6caffd640 1 -- 172.21.15.6:0/4241724198 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b4031870 con 0x7fe6cc0b6140 2024-03-18T14:43:13.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6caffd640 1 -- 172.21.15.6:0/4241724198 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4031c40 con 0x7fe6cc0b6140 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 -- 172.21.15.6:0/4241724198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 msgr2=0x7fe6cc0ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/4241724198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc0ceec0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc068ad0 tx=0x7fe6b402b910 comp rx=0 tx=0).stop 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 -- 172.21.15.6:0/4241724198 shutdown_connections 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/4241724198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/4241724198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc0ceec0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/4241724198 >> v2:172.21.15.92:3300/0 conn(0x7fe6cc15e030 0x7fe6cc0d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 -- 172.21.15.6:0/4241724198 >> 172.21.15.6:0/4241724198 conn(0x7fe6cc06c8c0 msgr2=0x7fe6cc06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 -- 172.21.15.6:0/4241724198 shutdown_connections 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.487+0000 7fe6d1c19640 1 -- 172.21.15.6:0/4241724198 wait complete. 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.489+0000 7fe6d1c19640 1 Processor -- start 2024-03-18T14:43:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.489+0000 7fe6d1c19640 1 -- start start 2024-03-18T14:43:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6d1c19640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6d1c19640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6d1c19640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6cc15e030 0x7fe6cc0d2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6d1c19640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc0ce550 con 0x7fe6cc0b6140 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6d1c19640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc0d3290 con 0x7fe6cc15e030 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6d1c19640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe6cc0ce830 con 0x7fe6cc0d0450 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc15ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54128/0 (socket says 172.21.15.6:54128) 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6cb7fe640 1 -- 172.21.15.6:0/435023838 learned_addr learned my addr 172.21.15.6:0/435023838 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.490+0000 7fe6c3fff640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cbfff640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6cc15e030 0x7fe6cc0d2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cb7fe640 1 -- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 msgr2=0x7fe6cc0d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cb7fe640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cb7fe640 1 -- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6cc15e030 msgr2=0x7fe6cc0d2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cb7fe640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6cc15e030 0x7fe6cc0d2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cb7fe640 1 -- 172.21.15.6:0/435023838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6cc15b530 con 0x7fe6cc0b6140 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cbfff640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6cc15e030 0x7fe6cc0d2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6cb7fe640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc15ab20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e810 tx=0x7fe6bc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6c8ff9640 1 -- 172.21.15.6:0/435023838 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc00f040 con 0x7fe6cc0b6140 2024-03-18T14:43:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6c8ff9640 1 -- 172.21.15.6:0/435023838 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6bc004030 con 0x7fe6cc0b6140 2024-03-18T14:43:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6cc1e7a30 con 0x7fe6cc0b6140 2024-03-18T14:43:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6c8ff9640 1 -- 172.21.15.6:0/435023838 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc004400 con 0x7fe6cc0b6140 2024-03-18T14:43:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.491+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6cc1e7ff0 con 0x7fe6cc0b6140 2024-03-18T14:43:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.492+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6cc06e1d0 con 0x7fe6cc0b6140 2024-03-18T14:43:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.493+0000 7fe6c8ff9640 1 -- 172.21.15.6:0/435023838 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe6bc00c6d0 con 0x7fe6cc0b6140 2024-03-18T14:43:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.493+0000 7fe6c8ff9640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe6ac06bbd0 0x7fe6ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.493+0000 7fe6c8ff9640 1 -- 172.21.15.6:0/435023838 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fe6bc08aad0 con 0x7fe6cc0b6140 2024-03-18T14:43:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.493+0000 7fe6c3fff640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe6ac06bbd0 0x7fe6ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.494+0000 7fe6c3fff640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe6ac06bbd0 0x7fe6ac06e090 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4036000 tx=0x7fe6b403c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.497+0000 7fe6c8ff9640 1 -- 172.21.15.6:0/435023838 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe6cc06e1d0 con 0x7fe6cc0b6140 2024-03-18T14:43:13.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.677+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe6cc0751b0 con 0x7fe6cc0b6140 2024-03-18T14:43:13.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.677+0000 7fe6c8ff9640 1 -- 172.21.15.6:0/435023838 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fe6cc15b530 con 0x7fe6cc0b6140 2024-03-18T14:43:13.679 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe6ac06bbd0 msgr2=0x7fe6ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe6ac06bbd0 0x7fe6ac06e090 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4036000 tx=0x7fe6b403c040 comp rx=0 tx=0).stop 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 msgr2=0x7fe6cc15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc15ab20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e810 tx=0x7fe6bc00ecd0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 shutdown_connections 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe6ac06bbd0 0x7fe6ac06e090 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6cc15e030 0x7fe6cc0d2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6cc0d0450 0x7fe6cc0d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 --2- 172.21.15.6:0/435023838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6cc0b6140 0x7fe6cc15ab20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 >> 172.21.15.6:0/435023838 conn(0x7fe6cc06c8c0 msgr2=0x7fe6cc15cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 shutdown_connections 2024-03-18T14:43:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.679+0000 7fe6d1c19640 1 -- 172.21.15.6:0/435023838 wait complete. 2024-03-18T14:43:13.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:13.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.799+0000 7f3c07e36640 1 Processor -- start 2024-03-18T14:43:13.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.799+0000 7f3c07e36640 1 -- start start 2024-03-18T14:43:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.799+0000 7f3c07e36640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3c000ce9d0 0x7f3c000cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.799+0000 7f3c07e36640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d2510 0x7f3c000d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.799+0000 7f3c07e36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c000d0560 0x7f3c0015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c07e36640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3c000d09d0 con 0x7f3c000d0560 2024-03-18T14:43:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c07e36640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f3c000cd8d0 con 0x7f3c000ce9d0 2024-03-18T14:43:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c07e36640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3c000d2970 con 0x7f3c000d2510 2024-03-18T14:43:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c000d0560 0x7f3c0015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3c000d0560 0x7f3c0015b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38102/0 (socket says 172.21.15.6:38102) 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 -- 172.21.15.6:0/1327355487 learned_addr learned my addr 172.21.15.6:0/1327355487 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c05bd2640 1 --2- 172.21.15.6:0/1327355487 >> v2:172.21.15.92:3300/0 conn(0x7f3c000ce9d0 0x7f3c000cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c053d1640 1 --2- 172.21.15.6:0/1327355487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d2510 0x7f3c000d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 -- 172.21.15.6:0/1327355487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d2510 msgr2=0x7f3c000d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 --2- 172.21.15.6:0/1327355487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d2510 0x7f3c000d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 -- 172.21.15.6:0/1327355487 >> v2:172.21.15.92:3300/0 conn(0x7f3c000ce9d0 msgr2=0x7f3c000cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 --2- 172.21.15.6:0/1327355487 >> v2:172.21.15.92:3300/0 conn(0x7f3c000ce9d0 0x7f3c000cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c063d3640 1 -- 172.21.15.6:0/1327355487 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c000cfac0 con 0x7f3c000d0560 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.800+0000 7f3c053d1640 1 --2- 172.21.15.6:0/1327355487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d2510 0x7f3c000d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c063d3640 1 --2- 172.21.15.6:0/1327355487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000d0560 0x7f3c0015b690 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3c00073720 tx=0x7f3bf802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889e78709572b9b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c04bd0640 1 -- 172.21.15.6:0/1327355487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80317d0 con 0x7f3c000d0560 2024-03-18T14:43:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c04bd0640 1 -- 172.21.15.6:0/1327355487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf8037070 con 0x7f3c000d0560 2024-03-18T14:43:13.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c04bd0640 1 -- 172.21.15.6:0/1327355487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8031b90 con 0x7f3c000d0560 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c07e36640 1 -- 172.21.15.6:0/1327355487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000d0560 msgr2=0x7f3c0015b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c07e36640 1 --2- 172.21.15.6:0/1327355487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000d0560 0x7f3c0015b690 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3c00073720 tx=0x7f3bf802b870 comp rx=0 tx=0).stop 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c07e36640 1 -- 172.21.15.6:0/1327355487 shutdown_connections 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c07e36640 1 --2- 172.21.15.6:0/1327355487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000d0560 0x7f3c0015b690 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c07e36640 1 --2- 172.21.15.6:0/1327355487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d2510 0x7f3c000d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c07e36640 1 --2- 172.21.15.6:0/1327355487 >> v2:172.21.15.92:3300/0 conn(0x7f3c000ce9d0 0x7f3c000cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.801+0000 7f3c07e36640 1 -- 172.21.15.6:0/1327355487 >> 172.21.15.6:0/1327355487 conn(0x7f3c0008b0c0 msgr2=0x7f3c000d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.802+0000 7f3c07e36640 1 -- 172.21.15.6:0/1327355487 shutdown_connections 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.802+0000 7f3c07e36640 1 -- 172.21.15.6:0/1327355487 wait complete. 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 Processor -- start 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 -- start start 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000ce9d0 0x7f3c0014bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d0560 0x7f3c001f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c000d2510 0x7f3c001f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3c000cd280 con 0x7f3c000ce9d0 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3c000ccfa0 con 0x7f3c000d2510 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.803+0000 7f3c07e36640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3c000cd560 con 0x7f3c000d0560 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000ce9d0 0x7f3c0014bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000ce9d0 0x7f3c0014bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38104/0 (socket says 172.21.15.6:38104) 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 -- 172.21.15.6:0/2432547038 learned_addr learned my addr 172.21.15.6:0/2432547038 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c063d3640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c000d2510 0x7f3c001f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c053d1640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d0560 0x7f3c001f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 -- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d0560 msgr2=0x7f3c001f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d0560 0x7f3c001f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 -- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c000d2510 msgr2=0x7f3c001f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c000d2510 0x7f3c001f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 -- 172.21.15.6:0/2432547038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c000c52f0 con 0x7f3c000ce9d0 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c05bd2640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000ce9d0 0x7f3c0014bd80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00cd40 tx=0x7f3bfc00a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3bf67fc640 1 -- 172.21.15.6:0/2432547038 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc003b90 con 0x7f3c000ce9d0 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3bf67fc640 1 -- 172.21.15.6:0/2432547038 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc003d30 con 0x7f3c000ce9d0 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c000c55d0 con 0x7f3c000ce9d0 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3bf67fc640 1 -- 172.21.15.6:0/2432547038 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc004260 con 0x7f3c000ce9d0 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.804+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c000c5b90 con 0x7f3c000ce9d0 2024-03-18T14:43:13.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.805+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c000751b0 con 0x7f3c000ce9d0 2024-03-18T14:43:13.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.806+0000 7f3bf67fc640 1 -- 172.21.15.6:0/2432547038 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3bfc00d040 con 0x7f3c000ce9d0 2024-03-18T14:43:13.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.806+0000 7f3bf67fc640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3be406bbd0 0x7f3be406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:13.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.806+0000 7f3bf67fc640 1 -- 172.21.15.6:0/2432547038 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f3bfc08b300 con 0x7f3c000ce9d0 2024-03-18T14:43:13.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.806+0000 7f3c053d1640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3be406bbd0 0x7f3be406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:13.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.807+0000 7f3c053d1640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3be406bbd0 0x7f3be406e090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3c0015f4f0 tx=0x7f3bf0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:13.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.811+0000 7f3bf67fc640 1 -- 172.21.15.6:0/2432547038 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3bfc014030 con 0x7f3c000ce9d0 2024-03-18T14:43:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.991+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3c000cfe00 con 0x7f3c000ce9d0 2024-03-18T14:43:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.991+0000 7f3bf67fc640 1 -- 172.21.15.6:0/2432547038 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f3bfc008040 con 0x7f3c000ce9d0 2024-03-18T14:43:13.993 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3be406bbd0 msgr2=0x7f3be406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3be406bbd0 0x7f3be406e090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3c0015f4f0 tx=0x7f3bf0008040 comp rx=0 tx=0).stop 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000ce9d0 msgr2=0x7f3c0014bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000ce9d0 0x7f3c0014bd80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00cd40 tx=0x7f3bfc00a610 comp rx=0 tx=0).stop 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 shutdown_connections 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3be406bbd0 0x7f3be406e090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c000d2510 0x7f3c001f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c000d0560 0x7f3c001f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 --2- 172.21.15.6:0/2432547038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c000ce9d0 0x7f3c0014bd80 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 >> 172.21.15.6:0/2432547038 conn(0x7f3c0008b0c0 msgr2=0x7f3c000b08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 shutdown_connections 2024-03-18T14:43:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:13.993+0000 7f3c07e36640 1 -- 172.21.15.6:0/2432547038 wait complete. 2024-03-18T14:43:14.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.143+0000 7fccf0190640 1 Processor -- start 2024-03-18T14:43:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.144+0000 7fccf0190640 1 -- start start 2024-03-18T14:43:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fccf0190640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 0x7fcce806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fccf0190640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcce815c0c0 con 0x7fcce806b700 2024-03-18T14:43:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fccf0190640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcce80cf410 con 0x7fcce80cedf0 2024-03-18T14:43:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fccf0190640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcce80cfbf0 con 0x7fcce815bce0 2024-03-18T14:43:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fcced72b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcce806b700 0x7fcce80ce6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56194/0 (socket says 172.21.15.6:56194) 2024-03-18T14:43:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fcced72b640 1 -- 172.21.15.6:0/300455699 learned_addr learned my addr 172.21.15.6:0/300455699 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fccedf2c640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 0x7fcce806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1562596281 0 0) 0x7fcce815c0c0 con 0x7fcce806b700 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.145+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd4003520 con 0x7fcce806b700 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2513541987 0 0) 0x7fccd4003520 con 0x7fcce806b700 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 msgr2=0x7fcce806b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 0x7fcce806b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 >> v1:172.21.15.92:6789/0 conn(0x7fcce80cedf0 legacy=0x7fcce80d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce80cf6f0 con 0x7fcce806b700 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fccdc001670 con 0x7fcce806b700 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 >> v1:172.21.15.6:6789/0 conn(0x7fcce806b700 legacy=0x7fcce80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 0x7fccd40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccd4003f60 0x7fccd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fccd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fccd400ba70 con 0x7fccd4004d40 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fccd400bc20 con 0x7fccd4003f60 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fccd400be20 con 0x7fcce815bce0 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccee72d640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fccd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccedf2c640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 0x7fccd40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fcced72b640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccd4003f60 0x7fccd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccee72d640 1 -- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 msgr2=0x7fccd40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccee72d640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce815bce0 0x7fccd40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccee72d640 1 -- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccd4003f60 msgr2=0x7fccd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccee72d640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccd4003f60 0x7fccd4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.146+0000 7fccee72d640 1 -- 172.21.15.6:0/300455699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd4003520 con 0x7fccd4004d40 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.147+0000 7fccee72d640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fccd40051c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcce0001e50 tx=0x7fccd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1ba02678f162ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.147+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce0001a70 con 0x7fccd4004d40 2024-03-18T14:43:14.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.147+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce0003b70 con 0x7fccd4004d40 2024-03-18T14:43:14.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.147+0000 7fccecf2a640 1 -- 172.21.15.6:0/300455699 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce00050d0 con 0x7fccd4004d40 2024-03-18T14:43:14.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.148+0000 7fccf0190640 1 -- 172.21.15.6:0/300455699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 msgr2=0x7fccd40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.148+0000 7fccf0190640 1 --2- 172.21.15.6:0/300455699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fccd40051c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcce0001e50 tx=0x7fccd4005830 comp rx=0 tx=0).stop 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.148+0000 7fccee72d640 1 -- 172.21.15.6:0/300455699 reap_dead start 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.148+0000 7fccf0190640 1 -- 172.21.15.6:0/300455699 shutdown_connections 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.148+0000 7fccf0190640 1 -- 172.21.15.6:0/300455699 >> 172.21.15.6:0/300455699 conn(0x7fcce8099a50 msgr2=0x7fcce8099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.148+0000 7fccf0190640 1 -- 172.21.15.6:0/300455699 shutdown_connections 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.148+0000 7fccf0190640 1 -- 172.21.15.6:0/300455699 wait complete. 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.149+0000 7fccf0190640 1 Processor -- start 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.149+0000 7fccf0190640 1 -- start start 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccf0190640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fcce80c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccf0190640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce81f8320 0x7fcce80b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccf0190640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcce806ebf0 0x7fcce80b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccf0190640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcce80d0190 con 0x7fccd4004d40 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccf0190640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcce806b1c0 con 0x7fcce806ebf0 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccf0190640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcce806b340 con 0x7fcce81f8320 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fcce80c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fcce80c3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38126/0 (socket says 172.21.15.6:38126) 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 -- 172.21.15.6:0/563136645 learned_addr learned my addr 172.21.15.6:0/563136645 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fcced72b640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce81f8320 0x7fcce80b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccee72d640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcce806ebf0 0x7fcce80b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 -- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce81f8320 msgr2=0x7fcce80b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce81f8320 0x7fcce80b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 -- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcce806ebf0 msgr2=0x7fcce80b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcce806ebf0 0x7fcce80b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.150+0000 7fccedf2c640 1 -- 172.21.15.6:0/563136645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce80b4df0 con 0x7fccd4004d40 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.151+0000 7fccedf2c640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fcce80c3730 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcce40119a0 tx=0x7fcce4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.151+0000 7fccda7fc640 1 -- 172.21.15.6:0/563136645 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce400faf0 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.151+0000 7fccda7fc640 1 -- 172.21.15.6:0/563136645 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce400fc90 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.151+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce80b5070 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.151+0000 7fccda7fc640 1 -- 172.21.15.6:0/563136645 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4014440 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.151+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce80b5630 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.152+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce80751b0 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.152+0000 7fccda7fc640 1 -- 172.21.15.6:0/563136645 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcce4012040 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.153+0000 7fccda7fc640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcccc06bc00 0x7fcccc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.153+0000 7fccda7fc640 1 -- 172.21.15.6:0/563136645 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fcce408b350 con 0x7fccd4004d40 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.153+0000 7fcced72b640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcccc06bc00 0x7fcccc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.153+0000 7fcced72b640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcccc06bc00 0x7fcccc06e0c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcce80cb210 tx=0x7fccdc000fa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.161+0000 7fccda7fc640 1 -- 172.21.15.6:0/563136645 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcce80b4df0 con 0x7fccd4004d40 2024-03-18T14:43:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.344+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcce80c4290 con 0x7fccd4004d40 2024-03-18T14:43:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.345+0000 7fccda7fc640 1 -- 172.21.15.6:0/563136645 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fcce4019030 con 0x7fccd4004d40 2024-03-18T14:43:14.347 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcccc06bc00 msgr2=0x7fcccc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcccc06bc00 0x7fcccc06e0c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcce80cb210 tx=0x7fccdc000fa0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 msgr2=0x7fcce80c3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fcce80c3730 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcce40119a0 tx=0x7fcce4011e60 comp rx=0 tx=0).stop 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 shutdown_connections 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcccc06bc00 0x7fcccc06e0c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcce806ebf0 0x7fcce80b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcce81f8320 0x7fcce80b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 --2- 172.21.15.6:0/563136645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccd4004d40 0x7fcce80c3730 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 >> 172.21.15.6:0/563136645 conn(0x7fcce8099a50 msgr2=0x7fcce8157000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 shutdown_connections 2024-03-18T14:43:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.347+0000 7fccf0190640 1 -- 172.21.15.6:0/563136645 wait complete. 2024-03-18T14:43:14.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-18T14:43:14.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.463+0000 7fd393ef7640 1 Processor -- start 2024-03-18T14:43:14.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.463+0000 7fd393ef7640 1 -- start start 2024-03-18T14:43:14.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd393ef7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd38c071ad0 0x7fd38c15cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd393ef7640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c071eb0 0x7fd38c0af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd393ef7640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38c0afef0 con 0x7fd38c15f850 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd393ef7640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd38c0b01d0 con 0x7fd38c071ad0 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd393ef7640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd38c0b04b0 con 0x7fd38c071eb0 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391c93640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd38c071ad0 0x7fd38c15cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391c93640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd38c071ad0 0x7fd38c15cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38140/0 (socket says 172.21.15.6:38140) 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391c93640 1 -- 172.21.15.6:0/3033941043 learned_addr learned my addr 172.21.15.6:0/3033941043 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391492640 1 --2- 172.21.15.6:0/3033941043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c071eb0 0x7fd38c0af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391c93640 1 -- 172.21.15.6:0/3033941043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c071eb0 msgr2=0x7fd38c0af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391c93640 1 --2- 172.21.15.6:0/3033941043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c071eb0 0x7fd38c0af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391c93640 1 -- 172.21.15.6:0/3033941043 >> v1:172.21.15.92:6789/0 conn(0x7fd38c15f850 legacy=0x7fd38c15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391c93640 1 -- 172.21.15.6:0/3033941043 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c15d5a0 con 0x7fd38c071ad0 2024-03-18T14:43:14.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.464+0000 7fd391492640 1 --2- 172.21.15.6:0/3033941043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c071eb0 0x7fd38c0af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:14.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.465+0000 7fd391c93640 1 --2- 172.21.15.6:0/3033941043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c15cf10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd388009870 tx=0x7fd38802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15c66f9cd6b1982 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.465+0000 7fd390c91640 1 -- 172.21.15.6:0/3033941043 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388035020 con 0x7fd38c071ad0 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.465+0000 7fd390c91640 1 -- 172.21.15.6:0/3033941043 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd388031640 con 0x7fd38c071ad0 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.465+0000 7fd390c91640 1 -- 172.21.15.6:0/3033941043 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38802d040 con 0x7fd38c071ad0 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.465+0000 7fd393ef7640 1 -- 172.21.15.6:0/3033941043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 msgr2=0x7fd38c15cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.465+0000 7fd393ef7640 1 --2- 172.21.15.6:0/3033941043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c15cf10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd388009870 tx=0x7fd38802b8f0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.466+0000 7fd393ef7640 1 -- 172.21.15.6:0/3033941043 shutdown_connections 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.466+0000 7fd393ef7640 1 --2- 172.21.15.6:0/3033941043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c071eb0 0x7fd38c0af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.466+0000 7fd393ef7640 1 --2- 172.21.15.6:0/3033941043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c15cf10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.466+0000 7fd393ef7640 1 -- 172.21.15.6:0/3033941043 >> 172.21.15.6:0/3033941043 conn(0x7fd38c06b1c0 msgr2=0x7fd38c06b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.466+0000 7fd393ef7640 1 -- 172.21.15.6:0/3033941043 shutdown_connections 2024-03-18T14:43:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.466+0000 7fd393ef7640 1 -- 172.21.15.6:0/3033941043 wait complete. 2024-03-18T14:43:14.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.467+0000 7fd393ef7640 1 Processor -- start 2024-03-18T14:43:14.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.467+0000 7fd393ef7640 1 -- start start 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c13b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd38c071eb0 0x7fd38c13bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c15f850 0x7fd38c143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c159be0 con 0x7fd38c071ad0 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c159a60 con 0x7fd38c071eb0 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd38c0afac0 con 0x7fd38c15f850 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c13b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c13b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38144/0 (socket says 172.21.15.6:38144) 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 -- 172.21.15.6:0/2158417817 learned_addr learned my addr 172.21.15.6:0/2158417817 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391492640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd38c071eb0 0x7fd38c13bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd392494640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c15f850 0x7fd38c143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 -- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c15f850 msgr2=0x7fd38c143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c15f850 0x7fd38c143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 -- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd38c071eb0 msgr2=0x7fd38c13bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd38c071eb0 0x7fd38c13bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 -- 172.21.15.6:0/2158417817 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c13c5b0 con 0x7fd38c071ad0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd391c93640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c13b420 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd388000c00 tx=0x7fd3880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd3827fc640 1 -- 172.21.15.6:0/2158417817 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388040070 con 0x7fd38c071ad0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd3827fc640 1 -- 172.21.15.6:0/2158417817 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd388039bf0 con 0x7fd38c071ad0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c13c890 con 0x7fd38c071ad0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd3827fc640 1 -- 172.21.15.6:0/2158417817 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388035040 con 0x7fd38c071ad0 2024-03-18T14:43:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.469+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c13d3c0 con 0x7fd38c071ad0 2024-03-18T14:43:14.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.470+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c0751b0 con 0x7fd38c071ad0 2024-03-18T14:43:14.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.471+0000 7fd3827fc640 1 -- 172.21.15.6:0/2158417817 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd38804b060 con 0x7fd38c071ad0 2024-03-18T14:43:14.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.471+0000 7fd3827fc640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd37006bbd0 0x7fd37006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.471+0000 7fd3827fc640 1 -- 172.21.15.6:0/2158417817 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7fd3880ac880 con 0x7fd38c071ad0 2024-03-18T14:43:14.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.472+0000 7fd391492640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd37006bbd0 0x7fd37006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.472+0000 7fd391492640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd37006bbd0 0x7fd37006e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0aec60 tx=0x7fd37c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.476+0000 7fd3827fc640 1 -- 172.21.15.6:0/2158417817 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd388031610 con 0x7fd38c071ad0 2024-03-18T14:43:14.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.658+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fd38c0cbee0 con 0x7fd38c071ad0 2024-03-18T14:43:14.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.658+0000 7fd3827fc640 1 -- 172.21.15.6:0/2158417817 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) ==== 97+0+11 (secure 0 0 0) 0x7fd38807b060 con 0x7fd38c071ad0 2024-03-18T14:43:14.660 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd37006bbd0 msgr2=0x7fd37006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd37006bbd0 0x7fd37006e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0aec60 tx=0x7fd37c008040 comp rx=0 tx=0).stop 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 msgr2=0x7fd38c13b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c13b420 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd388000c00 tx=0x7fd3880046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 shutdown_connections 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd37006bbd0 0x7fd37006e090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd38c15f850 0x7fd38c143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd38c071eb0 0x7fd38c13bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 --2- 172.21.15.6:0/2158417817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd38c071ad0 0x7fd38c13b420 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 >> 172.21.15.6:0/2158417817 conn(0x7fd38c06b1c0 msgr2=0x7fd38c06f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:14.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 shutdown_connections 2024-03-18T14:43:14.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.661+0000 7fd393ef7640 1 -- 172.21.15.6:0/2158417817 wait complete. 2024-03-18T14:43:14.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:14.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.814+0000 7f99832d1640 1 Processor -- start 2024-03-18T14:43:14.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.814+0000 7f99832d1640 1 -- start start 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f99832d1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f997c15e010 0x7f997c0d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f99832d1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f99832d1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f997c06ca50 con 0x7f997c0b1f00 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f99832d1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f997c06ce10 con 0x7f997c15e010 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f99832d1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f997c06d0f0 con 0x7f997c06c4e0 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998106d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f997c15e010 0x7f997c0d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998106d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f997c15e010 0x7f997c0d0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38146/0 (socket says 172.21.15.6:38146) 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998106d640 1 -- 172.21.15.6:0/171114861 learned_addr learned my addr 172.21.15.6:0/171114861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998086c640 1 --2- 172.21.15.6:0/171114861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998106d640 1 -- 172.21.15.6:0/171114861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 msgr2=0x7f997c0b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998106d640 1 --2- 172.21.15.6:0/171114861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998106d640 1 -- 172.21.15.6:0/171114861 >> v1:172.21.15.92:6789/0 conn(0x7f997c0b1f00 legacy=0x7f997c0ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.815+0000 7f998106d640 1 -- 172.21.15.6:0/171114861 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c0b34f0 con 0x7f997c15e010 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.816+0000 7f998106d640 1 --2- 172.21.15.6:0/171114861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c15e010 0x7f997c0d0e70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9974009e50 tx=0x7f997402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d35fd8ef288666 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.816+0000 7f997bfff640 1 -- 172.21.15.6:0/171114861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997402c9e0 con 0x7f997c15e010 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.816+0000 7f997bfff640 1 -- 172.21.15.6:0/171114861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f997402cb80 con 0x7f997c15e010 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.816+0000 7f997bfff640 1 -- 172.21.15.6:0/171114861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740343f0 con 0x7f997c15e010 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 -- 172.21.15.6:0/171114861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c15e010 msgr2=0x7f997c0d0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 --2- 172.21.15.6:0/171114861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c15e010 0x7f997c0d0e70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9974009e50 tx=0x7f997402be40 comp rx=0 tx=0).stop 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 -- 172.21.15.6:0/171114861 shutdown_connections 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 --2- 172.21.15.6:0/171114861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 --2- 172.21.15.6:0/171114861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c15e010 0x7f997c0d0e70 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 -- 172.21.15.6:0/171114861 >> 172.21.15.6:0/171114861 conn(0x7f997c08b320 msgr2=0x7f997c08b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 -- 172.21.15.6:0/171114861 shutdown_connections 2024-03-18T14:43:14.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.817+0000 7f99832d1640 1 -- 172.21.15.6:0/171114861 wait complete. 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.818+0000 7f99832d1640 1 Processor -- start 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.818+0000 7f99832d1640 1 -- start start 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f99832d1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f99832d1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c0b1f00 0x7f997c152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f99832d1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f997c15e010 0x7f997c152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f99832d1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f997c15e570 con 0x7f997c0b1f00 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f99832d1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f997c15e3f0 con 0x7f997c15e010 2024-03-18T14:43:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f99832d1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f997c0b2390 con 0x7f997c06c4e0 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c0b1f00 0x7f997c152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c0b1f00 0x7f997c152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38148/0 (socket says 172.21.15.6:38148) 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 -- 172.21.15.6:0/214294862 learned_addr learned my addr 172.21.15.6:0/214294862 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998106d640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998186e640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f997c15e010 0x7f997c152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 -- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 msgr2=0x7f997c0c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 -- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f997c15e010 msgr2=0x7f997c152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f997c15e010 0x7f997c152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 -- 172.21.15.6:0/214294862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c1532e0 con 0x7f997c0b1f00 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998106d640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998186e640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f997c15e010 0x7f997c152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.819+0000 7f998086c640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c0b1f00 0x7f997c152860 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f997000d800 tx=0x7f997000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.820+0000 7f9979ffb640 1 -- 172.21.15.6:0/214294862 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997000b6c0 con 0x7f997c0b1f00 2024-03-18T14:43:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.820+0000 7f9979ffb640 1 -- 172.21.15.6:0/214294862 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9970019070 con 0x7f997c0b1f00 2024-03-18T14:43:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.820+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c0b5620 con 0x7f997c0b1f00 2024-03-18T14:43:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.820+0000 7f9979ffb640 1 -- 172.21.15.6:0/214294862 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997000ba80 con 0x7f997c0b1f00 2024-03-18T14:43:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.820+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c0b5be0 con 0x7f997c0b1f00 2024-03-18T14:43:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.821+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f997c0d2830 con 0x7f997c0b1f00 2024-03-18T14:43:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.821+0000 7f9979ffb640 1 -- 172.21.15.6:0/214294862 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f997000bc20 con 0x7f997c0b1f00 2024-03-18T14:43:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.822+0000 7f9979ffb640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f996806bbd0 0x7f996806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.822+0000 7f9979ffb640 1 -- 172.21.15.6:0/214294862 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f997008b270 con 0x7f997c0b1f00 2024-03-18T14:43:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.822+0000 7f998106d640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f996806bbd0 0x7f996806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.822+0000 7f998106d640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f996806bbd0 0x7f996806e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9974035000 tx=0x7f9974030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:14.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:14.826+0000 7f9979ffb640 1 -- 172.21.15.6:0/214294862 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f997c0d2830 con 0x7f997c0b1f00 2024-03-18T14:43:15.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.019+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f997c0b09a0 con 0x7f997c0b1f00 2024-03-18T14:43:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.019+0000 7f9979ffb640 1 -- 172.21.15.6:0/214294862 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) ==== 80+0+0 (secure 0 0 0) 0x7f997c0b09a0 con 0x7f997c0b1f00 2024-03-18T14:43:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f996806bbd0 msgr2=0x7f996806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f996806bbd0 0x7f996806e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9974035000 tx=0x7f9974030000 comp rx=0 tx=0).stop 2024-03-18T14:43:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c0b1f00 msgr2=0x7f997c152860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c0b1f00 0x7f997c152860 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f997000d800 tx=0x7f997000dcc0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 shutdown_connections 2024-03-18T14:43:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f996806bbd0 0x7f996806e090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f997c15e010 0x7f997c152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f997c0b1f00 0x7f997c152860 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 --2- 172.21.15.6:0/214294862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f997c06c4e0 0x7f997c0c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.021+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 >> 172.21.15.6:0/214294862 conn(0x7f997c08b320 msgr2=0x7f997c15c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.022+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 shutdown_connections 2024-03-18T14:43:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.022+0000 7f99832d1640 1 -- 172.21.15.6:0/214294862 wait complete. 2024-03-18T14:43:15.033 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:15.034 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:15.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:15.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.200+0000 7f6409cdc640 1 Processor -- start 2024-03-18T14:43:15.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.200+0000 7f6409cdc640 1 -- start start 2024-03-18T14:43:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6409cdc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f640415e020 0x7f6404162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6409cdc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64040d3210 0x7f64040d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6409cdc640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64040b9660 con 0x7f64040ced00 2024-03-18T14:43:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6409cdc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f64040d2270 con 0x7f64040d3210 2024-03-18T14:43:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6409cdc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f64040ce390 con 0x7f640415e020 2024-03-18T14:43:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6403fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64040d3210 0x7f64040d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6403fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64040d3210 0x7f64040d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38152/0 (socket says 172.21.15.6:38152) 2024-03-18T14:43:15.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6403fff640 1 -- 172.21.15.6:0/2737293115 learned_addr learned my addr 172.21.15.6:0/2737293115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:15.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f64037fe640 1 --2- 172.21.15.6:0/2737293115 >> v2:172.21.15.92:3300/0 conn(0x7f640415e020 0x7f6404162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6403fff640 1 -- 172.21.15.6:0/2737293115 >> v2:172.21.15.92:3300/0 conn(0x7f640415e020 msgr2=0x7f6404162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:15.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.201+0000 7f6403fff640 1 --2- 172.21.15.6:0/2737293115 >> v2:172.21.15.92:3300/0 conn(0x7f640415e020 0x7f6404162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.202+0000 7f6403fff640 1 -- 172.21.15.6:0/2737293115 >> v1:172.21.15.92:6789/0 conn(0x7f64040ced00 legacy=0x7f64040d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:15.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.202+0000 7f6403fff640 1 -- 172.21.15.6:0/2737293115 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64040b9900 con 0x7f64040d3210 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.202+0000 7f6403fff640 1 --2- 172.21.15.6:0/2737293115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040d3210 0x7f64040d3c30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f64040d1450 tx=0x7f63fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b2b5a6eca0b713 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.202+0000 7f64027fc640 1 -- 172.21.15.6:0/2737293115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02ca80 con 0x7f64040d3210 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.202+0000 7f64027fc640 1 -- 172.21.15.6:0/2737293115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63fc02cc20 con 0x7f64040d3210 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.202+0000 7f64027fc640 1 -- 172.21.15.6:0/2737293115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0343f0 con 0x7f64040d3210 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 -- 172.21.15.6:0/2737293115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040d3210 msgr2=0x7f64040d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 --2- 172.21.15.6:0/2737293115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040d3210 0x7f64040d3c30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f64040d1450 tx=0x7f63fc02bee0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 -- 172.21.15.6:0/2737293115 shutdown_connections 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 --2- 172.21.15.6:0/2737293115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040d3210 0x7f64040d3c30 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 --2- 172.21.15.6:0/2737293115 >> v2:172.21.15.92:3300/0 conn(0x7f640415e020 0x7f6404162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 -- 172.21.15.6:0/2737293115 >> 172.21.15.6:0/2737293115 conn(0x7f6404072ce0 msgr2=0x7f64040730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 -- 172.21.15.6:0/2737293115 shutdown_connections 2024-03-18T14:43:15.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.203+0000 7f6409cdc640 1 -- 172.21.15.6:0/2737293115 wait complete. 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.204+0000 7f6409cdc640 1 Processor -- start 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.204+0000 7f6409cdc640 1 -- start start 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6409cdc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040ced00 0x7f6404157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6409cdc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64040d3210 0x7f64041f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6409cdc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f640415e020 0x7f64040b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6409cdc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f64040cf2c0 con 0x7f64040ced00 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6409cdc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f64040ce890 con 0x7f64040d3210 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6409cdc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f64040cf440 con 0x7f640415e020 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f64037fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040ced00 0x7f6404157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f64037fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040ced00 0x7f6404157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38156/0 (socket says 172.21.15.6:38156) 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f64037fe640 1 -- 172.21.15.6:0/1745010618 learned_addr learned my addr 172.21.15.6:0/1745010618 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6403fff640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f640415e020 0x7f64040b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f6402ffd640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64040d3210 0x7f64041f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f64037fe640 1 -- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f640415e020 msgr2=0x7f64040b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f64037fe640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f640415e020 0x7f64040b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f64037fe640 1 -- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64040d3210 msgr2=0x7f64041f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.205+0000 7f64037fe640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64040d3210 0x7f64041f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f64037fe640 1 -- 172.21.15.6:0/1745010618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64041f50c0 con 0x7f64040ced00 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f6402ffd640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64040d3210 0x7f64041f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f64037fe640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040ced00 0x7f6404157490 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f63f400d5f0 tx=0x7f63f400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f63ebfff640 1 -- 172.21.15.6:0/1745010618 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f40125f0 con 0x7f64040ced00 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f63ebfff640 1 -- 172.21.15.6:0/1745010618 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63f4019070 con 0x7f64040ced00 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64041f53a0 con 0x7f64040ced00 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f63ebfff640 1 -- 172.21.15.6:0/1745010618 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f40129d0 con 0x7f64040ced00 2024-03-18T14:43:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.206+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64041f5c90 con 0x7f64040ced00 2024-03-18T14:43:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.207+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f640408afe0 con 0x7f64040ced00 2024-03-18T14:43:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.207+0000 7f63ebfff640 1 -- 172.21.15.6:0/1745010618 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f63f4012b70 con 0x7f64040ced00 2024-03-18T14:43:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.208+0000 7f63ebfff640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f63f006bbd0 0x7f63f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.208+0000 7f63ebfff640 1 -- 172.21.15.6:0/1745010618 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f63f408b090 con 0x7f64040ced00 2024-03-18T14:43:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.208+0000 7f6402ffd640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f63f006bbd0 0x7f63f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.209+0000 7f6402ffd640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f63f006bbd0 0x7f63f006e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f63f8007fd0 tx=0x7f63f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.212+0000 7f63ebfff640 1 -- 172.21.15.6:0/1745010618 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f63f4059900 con 0x7f64040ced00 2024-03-18T14:43:15.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.484+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f640404f620 con 0x7f64040ced00 2024-03-18T14:43:15.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.485+0000 7f63ebfff640 1 -- 172.21.15.6:0/1745010618 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+18658 (secure 0 0 0) 0x7f63f4059200 con 0x7f64040ced00 2024-03-18T14:43:15.487 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:15.487 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:06.352665+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f63f006bbd0 msgr2=0x7f63f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f63f006bbd0 0x7f63f006e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f63f8007fd0 tx=0x7f63f8009040 comp rx=0 tx=0).stop 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040ced00 msgr2=0x7f6404157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040ced00 0x7f6404157490 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f63f400d5f0 tx=0x7f63f400dab0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 shutdown_connections 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f63f006bbd0 0x7f63f006e090 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f640415e020 0x7f64040b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64040d3210 0x7f64041f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 --2- 172.21.15.6:0/1745010618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64040ced00 0x7f6404157490 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.487+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 >> 172.21.15.6:0/1745010618 conn(0x7f6404072ce0 msgr2=0x7f6404159560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.488+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 shutdown_connections 2024-03-18T14:43:15.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.488+0000 7f6409cdc640 1 -- 172.21.15.6:0/1745010618 wait complete. 2024-03-18T14:43:15.500 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:42:54.939648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:42:55.638120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-18T14:43:03.985735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:15.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.603+0000 7f02bbc9a640 1 Processor -- start 2024-03-18T14:43:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.603+0000 7f02bbc9a640 1 -- start start 2024-03-18T14:43:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.603+0000 7f02bbc9a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b415e030 0x7f02b40d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.603+0000 7f02bbc9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02b40b6140 0x7f02b40ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.603+0000 7f02bbc9a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b40b65a0 con 0x7f02b40d0450 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.603+0000 7f02bbc9a640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f02b415e410 con 0x7f02b40b6140 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.603+0000 7f02bbc9a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f02b40ce300 con 0x7f02b415e030 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9235640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02b40b6140 0x7f02b40ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9235640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02b40b6140 0x7f02b40ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38158/0 (socket says 172.21.15.6:38158) 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9235640 1 -- 172.21.15.6:0/3837168231 learned_addr learned my addr 172.21.15.6:0/3837168231 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9a36640 1 --2- 172.21.15.6:0/3837168231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b415e030 0x7f02b40d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9235640 1 -- 172.21.15.6:0/3837168231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b415e030 msgr2=0x7f02b40d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9235640 1 --2- 172.21.15.6:0/3837168231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b415e030 0x7f02b40d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9235640 1 -- 172.21.15.6:0/3837168231 >> v1:172.21.15.92:6789/0 conn(0x7f02b40d0450 legacy=0x7f02b40d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9235640 1 -- 172.21.15.6:0/3837168231 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b40d2fd0 con 0x7f02b40b6140 2024-03-18T14:43:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.604+0000 7f02b9a36640 1 --2- 172.21.15.6:0/3837168231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b415e030 0x7f02b40d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:15.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02b9235640 1 --2- 172.21.15.6:0/3837168231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b40ceec0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009870 tx=0x7f02a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c789544ef3dd3cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02b8a34640 1 -- 172.21.15.6:0/3837168231 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4034070 con 0x7f02b40b6140 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02b8a34640 1 -- 172.21.15.6:0/3837168231 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a4030840 con 0x7f02b40b6140 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02b8a34640 1 -- 172.21.15.6:0/3837168231 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4030c10 con 0x7f02b40b6140 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3837168231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 msgr2=0x7f02b40ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3837168231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b40ceec0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009870 tx=0x7f02a402b980 comp rx=0 tx=0).stop 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3837168231 shutdown_connections 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3837168231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b40ceec0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3837168231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b415e030 0x7f02b40d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.605+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3837168231 >> 172.21.15.6:0/3837168231 conn(0x7f02b406c8c0 msgr2=0x7f02b406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.606+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3837168231 shutdown_connections 2024-03-18T14:43:15.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.606+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3837168231 wait complete. 2024-03-18T14:43:15.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.607+0000 7f02bbc9a640 1 Processor -- start 2024-03-18T14:43:15.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.607+0000 7f02bbc9a640 1 -- start start 2024-03-18T14:43:15.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b415a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b40d0450 0x7f02b415a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b415e030 0x7f02b41e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f02b40ce830 con 0x7f02b40b6140 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02b40ce550 con 0x7f02b415e030 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f02b40cf400 con 0x7f02b40d0450 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b415a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b415a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38170/0 (socket says 172.21.15.6:38170) 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 -- 172.21.15.6:0/3175869331 learned_addr learned my addr 172.21.15.6:0/3175869331 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9235640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b40d0450 0x7f02b415a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02ba237640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b415e030 0x7f02b41e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 -- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b40d0450 msgr2=0x7f02b415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b40d0450 0x7f02b415a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 -- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b415e030 msgr2=0x7f02b41e9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b415e030 0x7f02b41e9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 -- 172.21.15.6:0/3175869331 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b41e8980 con 0x7f02b40b6140 2024-03-18T14:43:15.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9235640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b40d0450 0x7f02b415a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:15.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02b9a36640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b415a740 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f02b000e7e0 tx=0x7f02b000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02aa7fc640 1 -- 172.21.15.6:0/3175869331 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b000f040 con 0x7f02b40b6140 2024-03-18T14:43:15.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02aa7fc640 1 -- 172.21.15.6:0/3175869331 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b0004030 con 0x7f02b40b6140 2024-03-18T14:43:15.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b41e8c60 con 0x7f02b40b6140 2024-03-18T14:43:15.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02aa7fc640 1 -- 172.21.15.6:0/3175869331 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b0004400 con 0x7f02b40b6140 2024-03-18T14:43:15.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.608+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b41e9220 con 0x7f02b40b6140 2024-03-18T14:43:15.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.609+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b40751b0 con 0x7f02b40b6140 2024-03-18T14:43:15.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.610+0000 7f02aa7fc640 1 -- 172.21.15.6:0/3175869331 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f02b000c6d0 con 0x7f02b40b6140 2024-03-18T14:43:15.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.610+0000 7f02aa7fc640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f029806bbd0 0x7f029806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.611+0000 7f02aa7fc640 1 -- 172.21.15.6:0/3175869331 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f02b008abf0 con 0x7f02b40b6140 2024-03-18T14:43:15.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.611+0000 7f02b9235640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f029806bbd0 0x7f029806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.611+0000 7f02b9235640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f029806bbd0 0x7f029806e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f02a4036000 tx=0x7f02a403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.615+0000 7f02aa7fc640 1 -- 172.21.15.6:0/3175869331 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f02b0017020 con 0x7f02b40b6140 2024-03-18T14:43:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.795+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f02b40d19f0 con 0x7f02b40b6140 2024-03-18T14:43:15.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.795+0000 7f02aa7fc640 1 -- 172.21.15.6:0/3175869331 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f02b008a020 con 0x7f02b40b6140 2024-03-18T14:43:15.796 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:15.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f029806bbd0 msgr2=0x7f029806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f029806bbd0 0x7f029806e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f02a4036000 tx=0x7f02a403b040 comp rx=0 tx=0).stop 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 msgr2=0x7f02b415a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b415a740 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f02b000e7e0 tx=0x7f02b000eca0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 shutdown_connections 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f029806bbd0 0x7f029806e090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b415e030 0x7f02b41e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b40d0450 0x7f02b415a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 --2- 172.21.15.6:0/3175869331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40b6140 0x7f02b415a740 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 >> 172.21.15.6:0/3175869331 conn(0x7f02b406c8c0 msgr2=0x7f02b40b6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 shutdown_connections 2024-03-18T14:43:15.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.797+0000 7f02bbc9a640 1 -- 172.21.15.6:0/3175869331 wait complete. 2024-03-18T14:43:15.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:15.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.926+0000 7f048fdf4640 1 Processor -- start 2024-03-18T14:43:15.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.926+0000 7f048fdf4640 1 -- start start 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.927+0000 7f048fdf4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04880d3570 0x7f04880b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.927+0000 7f048fdf4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04880d39d0 con 0x7f048815e030 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.927+0000 7f048fdf4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04880d2fd0 con 0x7f04880d0450 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.927+0000 7f048fdf4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f048815e410 con 0x7f04880d3570 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.927+0000 7f048d38f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04880d3570 0x7f04880b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.927+0000 7f048db90640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f048815e030 0x7f04880ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56206/0 (socket says 172.21.15.6:56206) 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.927+0000 7f048db90640 1 -- 172.21.15.6:0/4272755133 learned_addr learned my addr 172.21.15.6:0/4272755133 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.928+0000 7f048d38f640 1 -- 172.21.15.6:0/4272755133 >> v1:172.21.15.92:6789/0 conn(0x7f04880d0450 legacy=0x7f04880d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.928+0000 7f048d38f640 1 -- 172.21.15.6:0/4272755133 >> v1:172.21.15.6:6789/0 conn(0x7f048815e030 legacy=0x7f04880ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.928+0000 7f048d38f640 1 -- 172.21.15.6:0/4272755133 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04880d3290 con 0x7f04880d3570 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.928+0000 7f048d38f640 1 --2- 172.21.15.6:0/4272755133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04880d3570 0x7f04880b6140 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f04780097a0 tx=0x7f047802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6209ac534c6ef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.928+0000 7f048cb8e640 1 -- 172.21.15.6:0/4272755133 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478035020 con 0x7f04880d3570 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048cb8e640 1 -- 172.21.15.6:0/4272755133 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0478031830 con 0x7f04880d3570 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048cb8e640 1 -- 172.21.15.6:0/4272755133 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478031c00 con 0x7f04880d3570 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048fdf4640 1 -- 172.21.15.6:0/4272755133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04880d3570 msgr2=0x7f04880b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4272755133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04880d3570 0x7f04880b6140 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f04780097a0 tx=0x7f047802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048fdf4640 1 -- 172.21.15.6:0/4272755133 shutdown_connections 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4272755133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04880d3570 0x7f04880b6140 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048fdf4640 1 -- 172.21.15.6:0/4272755133 >> 172.21.15.6:0/4272755133 conn(0x7f048806c8c0 msgr2=0x7f048806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048fdf4640 1 -- 172.21.15.6:0/4272755133 shutdown_connections 2024-03-18T14:43:15.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.929+0000 7f048fdf4640 1 -- 172.21.15.6:0/4272755133 wait complete. 2024-03-18T14:43:15.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.930+0000 7f048fdf4640 1 Processor -- start 2024-03-18T14:43:15.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.930+0000 7f048fdf4640 1 -- start start 2024-03-18T14:43:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048fdf4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04880d0450 0x7f04880ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048fdf4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880d3570 0x7f04880ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048fdf4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f048815e030 0x7f048814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048fdf4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f04880ce830 con 0x7f048815e030 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048fdf4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f04880ce300 con 0x7f04880d3570 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048fdf4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0488161a90 con 0x7f04880d0450 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f048815e030 0x7f048814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f048815e030 0x7f048814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38186/0 (socket says 172.21.15.6:38186) 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 -- 172.21.15.6:0/4065121790 learned_addr learned my addr 172.21.15.6:0/4065121790 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048db90640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04880d0450 0x7f04880ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048d38f640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880d3570 0x7f04880ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 -- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04880d0450 msgr2=0x7f04880ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04880d0450 0x7f04880ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 -- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880d3570 msgr2=0x7f04880ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880d3570 0x7f04880ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 -- 172.21.15.6:0/4065121790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0488161390 con 0x7f048815e030 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.931+0000 7f048e391640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f048815e030 0x7f048814fe10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f048000c780 tx=0x7f048000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.932+0000 7f047e7fc640 1 -- 172.21.15.6:0/4065121790 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480017070 con 0x7f048815e030 2024-03-18T14:43:15.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.932+0000 7f047e7fc640 1 -- 172.21.15.6:0/4065121790 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0480003da0 con 0x7f048815e030 2024-03-18T14:43:15.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.932+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04881602f0 con 0x7f048815e030 2024-03-18T14:43:15.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.932+0000 7f047e7fc640 1 -- 172.21.15.6:0/4065121790 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04800052b0 con 0x7f048815e030 2024-03-18T14:43:15.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.932+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04881608b0 con 0x7f048815e030 2024-03-18T14:43:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.933+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04880751b0 con 0x7f048815e030 2024-03-18T14:43:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.933+0000 7f047e7fc640 1 -- 172.21.15.6:0/4065121790 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f048001a070 con 0x7f048815e030 2024-03-18T14:43:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.933+0000 7f047e7fc640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f046c06bbd0 0x7f046c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.934+0000 7f047e7fc640 1 -- 172.21.15.6:0/4065121790 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f048008a410 con 0x7f048815e030 2024-03-18T14:43:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.934+0000 7f048db90640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f046c06bbd0 0x7f046c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:15.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.934+0000 7f048db90640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f046c06bbd0 0x7f046c06e090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0484000f60 tx=0x7f0484009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:15.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:15.938+0000 7f047e7fc640 1 -- 172.21.15.6:0/4065121790 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0480058bf0 con 0x7f048815e030 2024-03-18T14:43:16.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.120+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f04880d21b0 con 0x7f048815e030 2024-03-18T14:43:16.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.120+0000 7f047e7fc640 1 -- 172.21.15.6:0/4065121790 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f04800584f0 con 0x7f048815e030 2024-03-18T14:43:16.121 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f046c06bbd0 msgr2=0x7f046c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f046c06bbd0 0x7f046c06e090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0484000f60 tx=0x7f0484009040 comp rx=0 tx=0).stop 2024-03-18T14:43:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f048815e030 msgr2=0x7f048814fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f048815e030 0x7f048814fe10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f048000c780 tx=0x7f048000cc40 comp rx=0 tx=0).stop 2024-03-18T14:43:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 shutdown_connections 2024-03-18T14:43:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f046c06bbd0 0x7f046c06e090 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f048815e030 0x7f048814fe10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880d3570 0x7f04880ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 --2- 172.21.15.6:0/4065121790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04880d0450 0x7f04880ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 >> 172.21.15.6:0/4065121790 conn(0x7f048806c8c0 msgr2=0x7f04880d08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 shutdown_connections 2024-03-18T14:43:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.122+0000 7f048fdf4640 1 -- 172.21.15.6:0/4065121790 wait complete. 2024-03-18T14:43:16.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:16.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.275+0000 7f459819d640 1 Processor -- start 2024-03-18T14:43:16.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.276+0000 7f459819d640 1 -- start start 2024-03-18T14:43:16.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.276+0000 7f459819d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f459015e030 0x7f45900d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.276+0000 7f459819d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45900b65a0 con 0x7f45900d0450 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.276+0000 7f459819d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45900d2fd0 con 0x7f45900b6140 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.276+0000 7f459819d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f459015e410 con 0x7f459015e030 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f459673a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f45900d0450 0x7f45900d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56208/0 (socket says 172.21.15.6:56208) 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f459673a640 1 -- 172.21.15.6:0/1992594039 learned_addr learned my addr 172.21.15.6:0/1992594039 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4595f39640 1 --2- 172.21.15.6:0/1992594039 >> v2:172.21.15.92:3300/0 conn(0x7f459015e030 0x7f45900d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159638018 0 0) 0x7f45900b65a0 con 0x7f45900d0450 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f457c003520 con 0x7f45900d0450 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813693485 0 0) 0x7f45900d2fd0 con 0x7f45900b6140 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45900b65a0 con 0x7f45900b6140 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4595f39640 1 -- 172.21.15.6:0/1992594039 >> v1:172.21.15.92:6789/0 conn(0x7f45900b6140 legacy=0x7f45900ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4595f39640 1 -- 172.21.15.6:0/1992594039 >> v1:172.21.15.6:6789/0 conn(0x7f45900d0450 legacy=0x7f45900d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3743841502 0 0) 0x7f457c003520 con 0x7f45900d0450 2024-03-18T14:43:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.277+0000 7f4595f39640 1 -- 172.21.15.6:0/1992594039 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45900d3290 con 0x7f459015e030 2024-03-18T14:43:16.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.278+0000 7f4595f39640 1 --2- 172.21.15.6:0/1992594039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459015e030 0x7f45900d3570 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f458c0097a0 tx=0x7f458c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ebfb9daffc1e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c038070 con 0x7f459015e030 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f458c031500 con 0x7f459015e030 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f4594f37640 1 -- 172.21.15.6:0/1992594039 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45880019f0 con 0x7f459015e030 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f459819d640 1 -- 172.21.15.6:0/1992594039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459015e030 msgr2=0x7f45900d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f459819d640 1 --2- 172.21.15.6:0/1992594039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459015e030 0x7f45900d3570 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f458c0097a0 tx=0x7f458c02b820 comp rx=0 tx=0).stop 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f459819d640 1 -- 172.21.15.6:0/1992594039 shutdown_connections 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f459819d640 1 --2- 172.21.15.6:0/1992594039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459015e030 0x7f45900d3570 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f459819d640 1 -- 172.21.15.6:0/1992594039 >> 172.21.15.6:0/1992594039 conn(0x7f459006c8c0 msgr2=0x7f459006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f459819d640 1 -- 172.21.15.6:0/1992594039 shutdown_connections 2024-03-18T14:43:16.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.279+0000 7f459819d640 1 -- 172.21.15.6:0/1992594039 wait complete. 2024-03-18T14:43:16.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.281+0000 7f459819d640 1 Processor -- start 2024-03-18T14:43:16.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.281+0000 7f459819d640 1 -- start start 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f459819d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45900b6140 0x7f459014cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f459819d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45900d0450 0x7f45900b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f459819d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459015e030 0x7f45900cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f459819d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f45900ce830 con 0x7f45900d0450 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f459819d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f45900ce300 con 0x7f45900b6140 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f459819d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f4590161a90 con 0x7f459015e030 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f4595738640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45900d0450 0x7f45900b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f4595738640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45900d0450 0x7f45900b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38202/0 (socket says 172.21.15.6:38202) 2024-03-18T14:43:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.282+0000 7f4595738640 1 -- 172.21.15.6:0/1139968211 learned_addr learned my addr 172.21.15.6:0/1139968211 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f459673a640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459015e030 0x7f45900cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595f39640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45900b6140 0x7f459014cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595738640 1 -- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459015e030 msgr2=0x7f45900cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595738640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459015e030 0x7f45900cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595738640 1 -- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45900b6140 msgr2=0x7f459014cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595738640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45900b6140 0x7f459014cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595738640 1 -- 172.21.15.6:0/1139968211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45900c7c50 con 0x7f45900d0450 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595f39640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45900b6140 0x7f459014cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f459673a640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459015e030 0x7f45900cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f4595738640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45900d0450 0x7f45900b43b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f458000c490 tx=0x7f458000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f45867fc640 1 -- 172.21.15.6:0/1139968211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800052d0 con 0x7f45900d0450 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f45867fc640 1 -- 172.21.15.6:0/1139968211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4580005470 con 0x7f45900d0450 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45900b4e50 con 0x7f45900d0450 2024-03-18T14:43:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f45867fc640 1 -- 172.21.15.6:0/1139968211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580012850 con 0x7f45900d0450 2024-03-18T14:43:16.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.283+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45900c7ef0 con 0x7f45900d0450 2024-03-18T14:43:16.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.284+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45900751b0 con 0x7f45900d0450 2024-03-18T14:43:16.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.285+0000 7f45867fc640 1 -- 172.21.15.6:0/1139968211 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f45800129f0 con 0x7f45900d0450 2024-03-18T14:43:16.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.285+0000 7f45867fc640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f457406bbd0 0x7f457406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.285+0000 7f45867fc640 1 -- 172.21.15.6:0/1139968211 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f458008aa70 con 0x7f45900d0450 2024-03-18T14:43:16.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.285+0000 7f4595f39640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f457406bbd0 0x7f457406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.286+0000 7f4595f39640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f457406bbd0 0x7f457406e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f458c035000 tx=0x7f458c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.289+0000 7f45867fc640 1 -- 172.21.15.6:0/1139968211 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f45800591c0 con 0x7f45900d0450 2024-03-18T14:43:16.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.499+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f45900d19f0 con 0x7f45900d0450 2024-03-18T14:43:16.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.500+0000 7f45867fc640 1 -- 172.21.15.6:0/1139968211 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f4580058ac0 con 0x7f45900d0450 2024-03-18T14:43:16.501 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f457406bbd0 msgr2=0x7f457406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f457406bbd0 0x7f457406e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f458c035000 tx=0x7f458c008030 comp rx=0 tx=0).stop 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45900d0450 msgr2=0x7f45900b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45900d0450 0x7f45900b43b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f458000c490 tx=0x7f458000c950 comp rx=0 tx=0).stop 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 shutdown_connections 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f457406bbd0 0x7f457406e090 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459015e030 0x7f45900cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45900d0450 0x7f45900b43b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 --2- 172.21.15.6:0/1139968211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45900b6140 0x7f459014cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 >> 172.21.15.6:0/1139968211 conn(0x7f459006c8c0 msgr2=0x7f45900b8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 shutdown_connections 2024-03-18T14:43:16.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.502+0000 7f459819d640 1 -- 172.21.15.6:0/1139968211 wait complete. 2024-03-18T14:43:16.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-18T14:43:16.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.622+0000 7f96296f1640 1 Processor -- start 2024-03-18T14:43:16.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.622+0000 7f96296f1640 1 -- start start 2024-03-18T14:43:16.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96296f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96240b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96296f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f962415e3b0 con 0x7f96240d2e50 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96296f1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96240b8470 con 0x7f962415dfd0 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96296f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f96240b8750 con 0x7f96240b5e50 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f961bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f96240d2e50 0x7f96240b5740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56218/0 (socket says 172.21.15.6:56218) 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f961bfff640 1 -- 172.21.15.6:0/852946322 learned_addr learned my addr 172.21.15.6:0/852946322 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96237fe640 1 --2- 172.21.15.6:0/852946322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96240b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96227fc640 1 -- 172.21.15.6:0/852946322 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458722002 0 0) 0x7f962415e3b0 con 0x7f96240d2e50 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96227fc640 1 -- 172.21.15.6:0/852946322 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9610003520 con 0x7f96240d2e50 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96237fe640 1 -- 172.21.15.6:0/852946322 >> v1:172.21.15.92:6789/0 conn(0x7f962415dfd0 legacy=0x7f96240b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96227fc640 1 -- 172.21.15.6:0/852946322 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170650808 0 0) 0x7f96240b8470 con 0x7f962415dfd0 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96237fe640 1 -- 172.21.15.6:0/852946322 >> v1:172.21.15.6:6789/0 conn(0x7f96240d2e50 legacy=0x7f96240b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.623+0000 7f96237fe640 1 -- 172.21.15.6:0/852946322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96240b8a30 con 0x7f96240b5e50 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.624+0000 7f96237fe640 1 --2- 172.21.15.6:0/852946322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96240b7e00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f961c009dc0 tx=0x7f961c02be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9288416e14e095a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.624+0000 7f96227fc640 1 -- 172.21.15.6:0/852946322 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c039070 con 0x7f96240b5e50 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.624+0000 7f96227fc640 1 -- 172.21.15.6:0/852946322 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961c02cc30 con 0x7f96240b5e50 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.624+0000 7f96227fc640 1 -- 172.21.15.6:0/852946322 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c034440 con 0x7f96240b5e50 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.625+0000 7f96296f1640 1 -- 172.21.15.6:0/852946322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 msgr2=0x7f96240b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.625+0000 7f96296f1640 1 --2- 172.21.15.6:0/852946322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96240b7e00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f961c009dc0 tx=0x7f961c02be80 comp rx=0 tx=0).stop 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.625+0000 7f96296f1640 1 -- 172.21.15.6:0/852946322 shutdown_connections 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.625+0000 7f96296f1640 1 --2- 172.21.15.6:0/852946322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96240b7e00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.625+0000 7f96296f1640 1 -- 172.21.15.6:0/852946322 >> 172.21.15.6:0/852946322 conn(0x7f9624072d10 msgr2=0x7f9624073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.625+0000 7f96296f1640 1 -- 172.21.15.6:0/852946322 shutdown_connections 2024-03-18T14:43:16.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.625+0000 7f96296f1640 1 -- 172.21.15.6:0/852946322 wait complete. 2024-03-18T14:43:16.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.626+0000 7f96296f1640 1 Processor -- start 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.626+0000 7f96296f1640 1 -- start start 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96296f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96241585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96296f1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96240d2e50 0x7f96240aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96296f1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f962415dfd0 0x7f96240aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96296f1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9624168c60 con 0x7f96240d2e50 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96296f1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9624168ae0 con 0x7f962415dfd0 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96296f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9624168de0 con 0x7f96240b5e50 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96240d2e50 0x7f96240aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96240d2e50 0x7f96240aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38206/0 (socket says 172.21.15.6:38206) 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 -- 172.21.15.6:0/2015773996 learned_addr learned my addr 172.21.15.6:0/2015773996 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f9622ffd640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96241585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96237fe640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f962415dfd0 0x7f96240aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 -- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 msgr2=0x7f96241585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96241585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 -- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f962415dfd0 msgr2=0x7f96240aac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f962415dfd0 0x7f96240aac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 -- 172.21.15.6:0/2015773996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962415c280 con 0x7f96240d2e50 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f96237fe640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f962415dfd0 0x7f96240aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.627+0000 7f961bfff640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96240d2e50 0x7f96240aa6c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f960c00c430 tx=0x7f960c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.628+0000 7f961b7fe640 1 -- 172.21.15.6:0/2015773996 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c017070 con 0x7f96240d2e50 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.628+0000 7f961b7fe640 1 -- 172.21.15.6:0/2015773996 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f960c005d00 con 0x7f96240d2e50 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.628+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9624145d10 con 0x7f96240d2e50 2024-03-18T14:43:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.628+0000 7f961b7fe640 1 -- 172.21.15.6:0/2015773996 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c007310 con 0x7f96240d2e50 2024-03-18T14:43:16.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.628+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9624158b10 con 0x7f96240d2e50 2024-03-18T14:43:16.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.629+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96240751b0 con 0x7f96240d2e50 2024-03-18T14:43:16.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.629+0000 7f961b7fe640 1 -- 172.21.15.6:0/2015773996 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f960c01a070 con 0x7f96240d2e50 2024-03-18T14:43:16.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.630+0000 7f961b7fe640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f960406bc00 0x7f960406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.630+0000 7f961b7fe640 1 -- 172.21.15.6:0/2015773996 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f960c08a090 con 0x7f96240d2e50 2024-03-18T14:43:16.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.630+0000 7f9622ffd640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f960406bc00 0x7f960406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.630+0000 7f9622ffd640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f960406bc00 0x7f960406e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9614000f60 tx=0x7f9614009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.638+0000 7f961b7fe640 1 -- 172.21.15.6:0/2015773996 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f960c058290 con 0x7f96240d2e50 2024-03-18T14:43:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.818+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f96240cef80 con 0x7f96240d2e50 2024-03-18T14:43:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.818+0000 7f961b7fe640 1 -- 172.21.15.6:0/2015773996 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7f960c05c330 con 0x7f96240d2e50 2024-03-18T14:43:16.820 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f960406bc00 msgr2=0x7f960406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f960406bc00 0x7f960406e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9614000f60 tx=0x7f9614009040 comp rx=0 tx=0).stop 2024-03-18T14:43:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96240d2e50 msgr2=0x7f96240aa6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96240d2e50 0x7f96240aa6c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f960c00c430 tx=0x7f960c00c8f0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 shutdown_connections 2024-03-18T14:43:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f960406bc00 0x7f960406e0c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f962415dfd0 0x7f96240aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96240d2e50 0x7f96240aa6c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 --2- 172.21.15.6:0/2015773996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96240b5e50 0x7f96241585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 >> 172.21.15.6:0/2015773996 conn(0x7f9624072d10 msgr2=0x7f96240b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 shutdown_connections 2024-03-18T14:43:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.820+0000 7f96296f1640 1 -- 172.21.15.6:0/2015773996 wait complete. 2024-03-18T14:43:16.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:16.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.934+0000 7f311f06c640 1 Processor -- start 2024-03-18T14:43:16.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.934+0000 7f311f06c640 1 -- start start 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.934+0000 7f311f06c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f311815dfb0 0x7f31180ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.934+0000 7f311f06c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31180b6540 con 0x7f31180d35d0 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.934+0000 7f311f06c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31180d20e0 con 0x7f3118161a10 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.934+0000 7f311f06c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f311815e390 con 0x7f311815dfb0 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f311d609640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f31180d35d0 0x7f31180b5e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56226/0 (socket says 172.21.15.6:56226) 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f311d609640 1 -- 172.21.15.6:0/2647878639 learned_addr learned my addr 172.21.15.6:0/2647878639 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f311ce08640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f311815dfb0 0x7f31180ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3811359941 0 0) 0x7f31180b6540 con 0x7f31180d35d0 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3108003520 con 0x7f31180d35d0 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3683106560 0 0) 0x7f3108003520 con 0x7f31180d35d0 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f311815dfb0 msgr2=0x7f31180ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f311815dfb0 0x7f31180ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 >> v1:172.21.15.92:6789/0 conn(0x7f3118161a10 legacy=0x7f31180b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3118161f40 con 0x7f31180d35d0 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f31100019f0 con 0x7f31180d35d0 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 >> v1:172.21.15.6:6789/0 conn(0x7f31180d35d0 legacy=0x7f31180b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31080039e0 0x7f3108003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3108004390 0x7f3108004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3108005060 0x7f310800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f311815e390 con 0x7f31080039e0 2024-03-18T14:43:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f310800bf30 con 0x7f3108004390 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f310800c130 con 0x7f3108005060 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f311ce08640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f311815dfb0 0x7f31180ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f3117fff640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3108004390 0x7f3108004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f311d609640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3108005060 0x7f310800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.935+0000 7f311ce08640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31080039e0 0x7f3108003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.936+0000 7f311ce08640 1 -- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3108005060 msgr2=0x7f310800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.936+0000 7f311ce08640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3108005060 0x7f310800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.936+0000 7f311ce08640 1 -- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3108004390 msgr2=0x7f3108004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.936+0000 7f311ce08640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3108004390 0x7f3108004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.936+0000 7f311ce08640 1 -- 172.21.15.6:0/2647878639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3108003520 con 0x7f31080039e0 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.936+0000 7f311ce08640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31080039e0 0x7f3108003dc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3118067f00 tx=0x7f3108005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a312fce05ae66d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c0097d0 con 0x7f31080039e0 2024-03-18T14:43:16.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f310c004030 con 0x7f31080039e0 2024-03-18T14:43:16.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f31177fe640 1 -- 172.21.15.6:0/2647878639 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c004400 con 0x7f31080039e0 2024-03-18T14:43:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f311f06c640 1 -- 172.21.15.6:0/2647878639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31080039e0 msgr2=0x7f3108003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f311f06c640 1 --2- 172.21.15.6:0/2647878639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31080039e0 0x7f3108003dc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3118067f00 tx=0x7f3108005830 comp rx=0 tx=0).stop 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f311d609640 1 -- 172.21.15.6:0/2647878639 reap_dead start 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f311f06c640 1 -- 172.21.15.6:0/2647878639 shutdown_connections 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f311f06c640 1 -- 172.21.15.6:0/2647878639 >> 172.21.15.6:0/2647878639 conn(0x7f311806c850 msgr2=0x7f311806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f311f06c640 1 -- 172.21.15.6:0/2647878639 shutdown_connections 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.937+0000 7f311f06c640 1 -- 172.21.15.6:0/2647878639 wait complete. 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.938+0000 7f311f06c640 1 Processor -- start 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.938+0000 7f311f06c640 1 -- start start 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311f06c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31080039e0 0x7f3118160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311f06c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f31180c9d80 0x7f31180ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311f06c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31180cd900 0x7f31180cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311f06c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f31181627e0 con 0x7f31180cd900 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311f06c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3118162500 con 0x7f31080039e0 2024-03-18T14:43:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311f06c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f31180d1010 con 0x7f31180c9d80 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311d609640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31180cd900 0x7f31180cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311d609640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31180cd900 0x7f31180cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38212/0 (socket says 172.21.15.6:38212) 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.939+0000 7f311d609640 1 -- 172.21.15.6:0/3433585519 learned_addr learned my addr 172.21.15.6:0/3433585519 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311d609640 1 -- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f31180c9d80 msgr2=0x7f31180ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311d609640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f31180c9d80 0x7f31180ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311d609640 1 -- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31080039e0 msgr2=0x7f3118160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311d609640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31080039e0 0x7f3118160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311d609640 1 -- 172.21.15.6:0/3433585519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31180c8a00 con 0x7f31180cd900 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311d609640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31180cd900 0x7f31180cdd80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3110000f80 tx=0x7f3110007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f31157fa640 1 -- 172.21.15.6:0/3433585519 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311000ab80 con 0x7f31180cd900 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f31157fa640 1 -- 172.21.15.6:0/3433585519 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3110019070 con 0x7f31180cd900 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31180c8ce0 con 0x7f31180cd900 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f31157fa640 1 -- 172.21.15.6:0/3433585519 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31100133f0 con 0x7f31180cd900 2024-03-18T14:43:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.940+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31180c92a0 con 0x7f31180cd900 2024-03-18T14:43:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.942+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311804f5b0 con 0x7f31180cd900 2024-03-18T14:43:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.942+0000 7f31157fa640 1 -- 172.21.15.6:0/3433585519 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3110013590 con 0x7f31180cd900 2024-03-18T14:43:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.942+0000 7f31157fa640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f30fc06bbd0 0x7f30fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.942+0000 7f31157fa640 1 -- 172.21.15.6:0/3433585519 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f311008b330 con 0x7f31180cd900 2024-03-18T14:43:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.942+0000 7f311ce08640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f30fc06bbd0 0x7f30fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.943+0000 7f311ce08640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f30fc06bbd0 0x7f30fc06e090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f310c035000 tx=0x7f310c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:16.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:16.951+0000 7f31157fa640 1 -- 172.21.15.6:0/3433585519 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3110059af0 con 0x7f31180cd900 2024-03-18T14:43:17.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.271+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f311815bec0 con 0x7f31180cd900 2024-03-18T14:43:17.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.272+0000 7f31157fa640 1 -- 172.21.15.6:0/3433585519 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7f31100593f0 con 0x7f31180cd900 2024-03-18T14:43:17.273 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f30fc06bbd0 msgr2=0x7f30fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f30fc06bbd0 0x7f30fc06e090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f310c035000 tx=0x7f310c02e000 comp rx=0 tx=0).stop 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31180cd900 msgr2=0x7f31180cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31180cd900 0x7f31180cdd80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3110000f80 tx=0x7f3110007a90 comp rx=0 tx=0).stop 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 shutdown_connections 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f30fc06bbd0 0x7f30fc06e090 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31180cd900 0x7f31180cdd80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f31180c9d80 0x7f31180ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 --2- 172.21.15.6:0/3433585519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31080039e0 0x7f3118160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 >> 172.21.15.6:0/3433585519 conn(0x7f311806c850 msgr2=0x7f311815c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 shutdown_connections 2024-03-18T14:43:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.274+0000 7f311f06c640 1 -- 172.21.15.6:0/3433585519 wait complete. 2024-03-18T14:43:17.286 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:43:17.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:17.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:17.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[55729]: fuse finished with error 0 and tester_r 0 2024-03-18T14:43:17.317 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:17.406 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:43:17.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:17.406 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:43:17.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-18T14:43:17.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.568+0000 7fce582b2640 1 Processor -- start 2024-03-18T14:43:17.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.568+0000 7fce582b2640 1 -- start start 2024-03-18T14:43:17.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.569+0000 7fce582b2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce500ada90 0x7fce5015eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.569+0000 7fce582b2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce500aff50 con 0x7fce500ad620 2024-03-18T14:43:17.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.569+0000 7fce582b2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce500b1f60 con 0x7fce5015dfb0 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.569+0000 7fce582b2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fce500b0560 con 0x7fce500ada90 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.569+0000 7fce5584d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fce500ad620 0x7fce5015e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56242/0 (socket says 172.21.15.6:56242) 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.569+0000 7fce5584d640 1 -- 172.21.15.6:0/3682151097 learned_addr learned my addr 172.21.15.6:0/3682151097 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2893062295 0 0) 0x7fce500aff50 con 0x7fce500ad620 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce3c003520 con 0x7fce500ad620 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5684f640 1 --2- 172.21.15.6:0/3682151097 >> v2:172.21.15.92:3300/0 conn(0x7fce500ada90 0x7fce5015eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3409821229 0 0) 0x7fce3c003520 con 0x7fce500ad620 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 >> v2:172.21.15.92:3300/0 conn(0x7fce500ada90 msgr2=0x7fce5015eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 --2- 172.21.15.6:0/3682151097 >> v2:172.21.15.92:3300/0 conn(0x7fce500ada90 0x7fce5015eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 >> v1:172.21.15.92:6789/0 conn(0x7fce5015dfb0 legacy=0x7fce500b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce500b01f0 con 0x7fce500ad620 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fce4c001690 con 0x7fce500ad620 2024-03-18T14:43:17.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 >> v1:172.21.15.6:6789/0 conn(0x7fce500ad620 legacy=0x7fce5015e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce3c0039b0 0x7fce3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c0051c0 0x7fce3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fce500b0560 con 0x7fce3c0051c0 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c00c060 con 0x7fce3c004800 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fce3c00c260 con 0x7fce3c0039b0 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5684f640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c0051c0 0x7fce3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5584d640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.570+0000 7fce5604e640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce3c0039b0 0x7fce3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5684f640 1 -- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce3c0039b0 msgr2=0x7fce3c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5684f640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce3c0039b0 0x7fce3c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5684f640 1 -- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c004800 msgr2=0x7fce3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5684f640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c004800 0x7fce3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5684f640 1 -- 172.21.15.6:0/3682151097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3c003520 con 0x7fce3c0051c0 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5604e640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce3c0039b0 0x7fce3c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5684f640 1 -- 172.21.15.6:0/3682151097 reap_dead start 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5684f640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c0051c0 0x7fce3c00b9c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fce48009fc0 tx=0x7fce3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d622bab52b98451b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48009c10 con 0x7fce3c0051c0 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4802cca0 con 0x7fce3c0051c0 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.571+0000 7fce5504c640 1 -- 172.21.15.6:0/3682151097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48003bd0 con 0x7fce3c0051c0 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.572+0000 7fce582b2640 1 -- 172.21.15.6:0/3682151097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c0051c0 msgr2=0x7fce3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.572+0000 7fce582b2640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c0051c0 0x7fce3c00b9c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fce48009fc0 tx=0x7fce3c005830 comp rx=0 tx=0).stop 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.572+0000 7fce582b2640 1 -- 172.21.15.6:0/3682151097 shutdown_connections 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.572+0000 7fce582b2640 1 --2- 172.21.15.6:0/3682151097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce3c0051c0 0x7fce3c00b9c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.572+0000 7fce582b2640 1 -- 172.21.15.6:0/3682151097 >> 172.21.15.6:0/3682151097 conn(0x7fce500d09b0 msgr2=0x7fce500d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.572+0000 7fce582b2640 1 -- 172.21.15.6:0/3682151097 shutdown_connections 2024-03-18T14:43:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.572+0000 7fce582b2640 1 -- 172.21.15.6:0/3682151097 wait complete. 2024-03-18T14:43:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.573+0000 7fce582b2640 1 Processor -- start 2024-03-18T14:43:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.573+0000 7fce582b2640 1 -- start start 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce582b2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c0051c0 0x7fce500c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce582b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce50159d30 0x7fce50159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce582b2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce500c5940 0x7fce500b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce582b2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fce500afad0 con 0x7fce500c5940 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce582b2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fce500adec0 con 0x7fce3c0051c0 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce582b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fce500afc50 con 0x7fce50159d30 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce500c5940 0x7fce500b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce500c5940 0x7fce500b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38218/0 (socket says 172.21.15.6:38218) 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 -- 172.21.15.6:0/3619369399 learned_addr learned my addr 172.21.15.6:0/3619369399 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5584d640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce50159d30 0x7fce50159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5604e640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c0051c0 0x7fce500c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 -- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce50159d30 msgr2=0x7fce50159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce50159d30 0x7fce50159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 -- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c0051c0 msgr2=0x7fce500c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c0051c0 0x7fce500c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5684f640 1 -- 172.21.15.6:0/3619369399 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce500c48a0 con 0x7fce500c5940 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5584d640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce50159d30 0x7fce50159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.574+0000 7fce5604e640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c0051c0 0x7fce500c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.575+0000 7fce5684f640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce500c5940 0x7fce500b5a30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fce48035d50 tx=0x7fce48009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.575+0000 7fce42ffd640 1 -- 172.21.15.6:0/3619369399 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48003990 con 0x7fce500c5940 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.575+0000 7fce42ffd640 1 -- 172.21.15.6:0/3619369399 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce48044070 con 0x7fce500c5940 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.575+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce500c4b80 con 0x7fce500c5940 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.575+0000 7fce42ffd640 1 -- 172.21.15.6:0/3619369399 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48003d90 con 0x7fce500c5940 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.575+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce500c5140 con 0x7fce500c5940 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.576+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce500751b0 con 0x7fce500c5940 2024-03-18T14:43:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.576+0000 7fce42ffd640 1 -- 172.21.15.6:0/3619369399 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fce48005d10 con 0x7fce500c5940 2024-03-18T14:43:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.577+0000 7fce42ffd640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fce3406bc00 0x7fce3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.577+0000 7fce42ffd640 1 -- 172.21.15.6:0/3619369399 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7fce480ad530 con 0x7fce500c5940 2024-03-18T14:43:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.577+0000 7fce5604e640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fce3406bc00 0x7fce3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.578+0000 7fce5604e640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fce3406bc00 0x7fce3406e0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fce44007fd0 tx=0x7fce44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.581+0000 7fce42ffd640 1 -- 172.21.15.6:0/3619369399 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fce4807bcf0 con 0x7fce500c5940 2024-03-18T14:43:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.843+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7fce500af390 con 0x7fce500c5940 2024-03-18T14:43:17.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.845+0000 7fce42ffd640 1 -- 172.21.15.6:0/3619369399 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v13) ==== 211+0+168 (secure 0 0 0) 0x7fce4807b5f0 con 0x7fce500c5940 2024-03-18T14:43:17.846 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:43:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:43:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow rwp" 2024-03-18T14:43:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:43:17.847 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-18T14:43:17.847 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:43:17.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fce3406bc00 msgr2=0x7fce3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fce3406bc00 0x7fce3406e0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fce44007fd0 tx=0x7fce44009040 comp rx=0 tx=0).stop 2024-03-18T14:43:17.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce500c5940 msgr2=0x7fce500b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce500c5940 0x7fce500b5a30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fce48035d50 tx=0x7fce48009750 comp rx=0 tx=0).stop 2024-03-18T14:43:17.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 shutdown_connections 2024-03-18T14:43:17.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fce3406bc00 0x7fce3406e0c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce500c5940 0x7fce500b5a30 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce50159d30 0x7fce50159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 --2- 172.21.15.6:0/3619369399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce3c0051c0 0x7fce500c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.847+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 >> 172.21.15.6:0/3619369399 conn(0x7fce500d09b0 msgr2=0x7fce5006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.848+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 shutdown_connections 2024-03-18T14:43:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.848+0000 7fce582b2640 1 -- 172.21.15.6:0/3619369399 wait complete. 2024-03-18T14:43:17.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:43:17.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.963+0000 7f9fbe4fe640 1 Processor -- start 2024-03-18T14:43:17.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.963+0000 7f9fbe4fe640 1 -- start start 2024-03-18T14:43:17.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fbe4fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9fb815e020 0x7f9fb8162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fbe4fe640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb80b9660 con 0x7f9fb80ced00 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fbe4fe640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb80ce390 con 0x7f9fb80d3210 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fbe4fe640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9fb80d2270 con 0x7f9fb815e020 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9fb80ced00 0x7f9fb80d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56258/0 (socket says 172.21.15.6:56258) 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb77fe640 1 -- 172.21.15.6:0/1533448517 learned_addr learned my addr 172.21.15.6:0/1533448517 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 320359687 0 0) 0x7f9fb80b9660 con 0x7f9fb80ced00 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fa0003520 con 0x7f9fb80ced00 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb7fff640 1 --2- 172.21.15.6:0/1533448517 >> v2:172.21.15.92:3300/0 conn(0x7f9fb815e020 0x7f9fb8162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4056454980 0 0) 0x7f9fa0003520 con 0x7f9fb80ced00 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 >> v2:172.21.15.92:3300/0 conn(0x7f9fb815e020 msgr2=0x7f9fb8162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 --2- 172.21.15.6:0/1533448517 >> v2:172.21.15.92:3300/0 conn(0x7f9fb815e020 0x7f9fb8162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 >> v1:172.21.15.92:6789/0 conn(0x7f9fb80d3210 legacy=0x7f9fb80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb80b9900 con 0x7f9fb80ced00 2024-03-18T14:43:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f9fac001690 con 0x7f9fb80ced00 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 >> v1:172.21.15.6:6789/0 conn(0x7f9fb80ced00 legacy=0x7f9fb80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa00039b0 0x7f9fa0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa0004800 0x7f9fa0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fa00051c0 0x7f9fa000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb80d2270 con 0x7f9fa0004800 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa000c160 con 0x7f9fa00039b0 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa000c360 con 0x7f9fa00051c0 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.964+0000 7f9fb77fe640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa0004800 0x7f9fa0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fb7fff640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa00039b0 0x7f9fa0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fbca9b640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fa00051c0 0x7f9fa000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fb77fe640 1 -- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fa00051c0 msgr2=0x7f9fa000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fb77fe640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fa00051c0 0x7f9fa000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fb77fe640 1 -- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa00039b0 msgr2=0x7f9fa0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fb77fe640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa00039b0 0x7f9fa0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fb77fe640 1 -- 172.21.15.6:0/1533448517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa0003520 con 0x7f9fa0004800 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.965+0000 7f9fbca9b640 1 -- 172.21.15.6:0/1533448517 reap_dead start 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fb77fe640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa0004800 0x7f9fa0004c80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00c210 tx=0x7f9fa0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5984d4a0d9d3b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac007d30 con 0x7f9fa0004800 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fac030070 con 0x7f9fa0004800 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fb6ffd640 1 -- 172.21.15.6:0/1533448517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac005d30 con 0x7f9fa0004800 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/1533448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa0004800 msgr2=0x7f9fa0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa0004800 0x7f9fa0004c80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00c210 tx=0x7f9fa0005830 comp rx=0 tx=0).stop 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/1533448517 shutdown_connections 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/1533448517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa0004800 0x7f9fa0004c80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/1533448517 >> 172.21.15.6:0/1533448517 conn(0x7f9fb8072ce0 msgr2=0x7f9fb80730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/1533448517 shutdown_connections 2024-03-18T14:43:17.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.966+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/1533448517 wait complete. 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.968+0000 7f9fbe4fe640 1 Processor -- start 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.968+0000 7f9fbe4fe640 1 -- start start 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.969+0000 7f9fbe4fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa0004800 0x7f9fb8154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.969+0000 7f9fbe4fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fb815e570 0x7f9fb80d3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.969+0000 7f9fbe4fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fb8153400 0x7f9fb80c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.969+0000 7f9fbe4fe640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb80cf2c0 con 0x7f9fb815e570 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.969+0000 7f9fbe4fe640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb80ce890 con 0x7f9fa0004800 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.969+0000 7f9fbe4fe640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb80cf440 con 0x7f9fb8153400 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fb815e570 0x7f9fb80d3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fb815e570 0x7f9fb80d3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38228/0 (socket says 172.21.15.6:38228) 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 -- 172.21.15.6:0/3056300998 learned_addr learned my addr 172.21.15.6:0/3056300998 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb7fff640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa0004800 0x7f9fb8154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fbca9b640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fb8153400 0x7f9fb80c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 -- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fb8153400 msgr2=0x7f9fb80c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fb8153400 0x7f9fb80c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 -- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa0004800 msgr2=0x7f9fb8154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa0004800 0x7f9fb8154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 -- 172.21.15.6:0/3056300998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb8153e30 con 0x7f9fb815e570 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fbca9b640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fb8153400 0x7f9fb80c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb77fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fb815e570 0x7f9fb80d3ea0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9fac001990 tx=0x7f9fac02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb4ff9640 1 -- 172.21.15.6:0/3056300998 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac046070 con 0x7f9fb815e570 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb4ff9640 1 -- 172.21.15.6:0/3056300998 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fac000ed0 con 0x7f9fb815e570 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fb4ff9640 1 -- 172.21.15.6:0/3056300998 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac007d30 con 0x7f9fb815e570 2024-03-18T14:43:17.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb81540c0 con 0x7f9fb815e570 2024-03-18T14:43:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.970+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb8154960 con 0x7f9fb815e570 2024-03-18T14:43:17.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.971+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb808afe0 con 0x7f9fb815e570 2024-03-18T14:43:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.972+0000 7f9fb4ff9640 1 -- 172.21.15.6:0/3056300998 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9fac007070 con 0x7f9fb815e570 2024-03-18T14:43:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.972+0000 7f9fb4ff9640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9fa406bbd0 0x7f9fa406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.972+0000 7f9fb4ff9640 1 -- 172.21.15.6:0/3056300998 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f9fac0abf40 con 0x7f9fb815e570 2024-03-18T14:43:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.972+0000 7f9fb7fff640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9fa406bbd0 0x7f9fa406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:17.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.973+0000 7f9fb7fff640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9fa406bbd0 0x7f9fa406e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9fb815c810 tx=0x7f9fa8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:17.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:17.977+0000 7f9fb4ff9640 1 -- 172.21.15.6:0/3056300998 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9fac07a0b0 con 0x7f9fb815e570 2024-03-18T14:43:18.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.198+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9fb804f620 con 0x7f9fb815e570 2024-03-18T14:43:18.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.198+0000 7f9fb4ff9640 1 -- 172.21.15.6:0/3056300998 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+82 (secure 0 0 0) 0x7f9fac07e150 con 0x7f9fb815e570 2024-03-18T14:43:18.199 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9fa406bbd0 msgr2=0x7f9fa406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9fa406bbd0 0x7f9fa406e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9fb815c810 tx=0x7f9fa8009190 comp rx=0 tx=0).stop 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fb815e570 msgr2=0x7f9fb80d3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fb815e570 0x7f9fb80d3ea0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9fac001990 tx=0x7f9fac02e070 comp rx=0 tx=0).stop 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 shutdown_connections 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9fa406bbd0 0x7f9fa406e090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fb8153400 0x7f9fb80c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fb815e570 0x7f9fb80d3ea0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 --2- 172.21.15.6:0/3056300998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fa0004800 0x7f9fb8154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 >> 172.21.15.6:0/3056300998 conn(0x7f9fb8072ce0 msgr2=0x7f9fb8161e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 shutdown_connections 2024-03-18T14:43:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18.200+0000 7f9fbe4fe640 1 -- 172.21.15.6:0/3056300998 wait complete. 2024-03-18T14:43:18.212 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:43:18.212 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:43:18.212 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:43:18.212 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:43:18.213 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:18.213 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:43:18.213 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:43:18.213 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:43:18.213 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:18.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:18.213 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:43:18.228 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:43:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:43:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:43:18.228 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:43:18.229 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42567sec preferred_lft 42567sec 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:18.230 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:43:18.231 INFO:teuthology.orchestra.run.smithi006.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c41d:adff:fe29:e32/64 scope link 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::d035:11ff:fe58:6e4e/64 scope link 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::9023:d7ff:fecf:2a25/64 scope link 2024-03-18T14:43:18.232 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:18.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:18.233 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:43:18.294 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:43:18.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:18.296 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:43:18.351 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:43:18.351 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:43:18.351 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:43:18.352 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:43:18.352 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:43:18.352 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:43:18.352 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:43:18.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:43:18.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:43:18.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:43:18.352 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:43:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:43:18.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:18.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:18.496 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:43:18.510 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T14:43:18.510 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:43:18.511 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:18.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:18.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:43:18.577 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T14:43:18.577 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:18.577 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:18.631 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:18.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:18.632 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:18.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:18.754 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:18.809 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:18.810 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:18.811 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:18.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:18.812 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:18.865 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:43:18.865 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T14:43:18.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:18.935 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:18.992 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:18.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:18.993 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:18.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:2024-03-18T14:43:18.993+0000 7f69307ff480 -1 init, newargv = 0x5582af803080 newargc=15 2024-03-18T14:43:18.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[56794]: starting ceph client 2024-03-18T14:43:18.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[56794]: starting fuse 2024-03-18T14:43:19.009 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:43:19.009 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:43:19.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:43:19.010 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.0")) 2024-03-18T14:43:19.011 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:43:19.129 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.0.56794.asok 2024-03-18T14:43:19.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:19.141 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56794.asok 2024-03-18T14:43:19.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:19.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56794.asok status 2024-03-18T14:43:19.277 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:19.277 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "0", 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "56794", 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5970, 2024-03-18T14:43:19.278 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5970 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 361609801 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:19.279 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 361609801 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.5970 192.168.0.1:0/361609801", 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/361609801", 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 16, 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 44, 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:43:19.280 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:19.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:19.286 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:43:19.286 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:19.301 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:43:19.302 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:19.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:19.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:19.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:19.369 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-18T14:43:19.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:19.426 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-18T14:43:19.484 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:19.485 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:19.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:19.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.680+0000 7f7b24412640 1 Processor -- start 2024-03-18T14:43:19.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.680+0000 7f7b24412640 1 -- start start 2024-03-18T14:43:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b24412640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b1c15e020 0x7f7b1c162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b24412640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0ced00 0x7f7b1c0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b24412640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7b1c0d3210 0x7f7b1c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b24412640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7b1c0b9660 con 0x7f7b1c15e020 2024-03-18T14:43:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b24412640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7b1c0ce390 con 0x7f7b1c0d3210 2024-03-18T14:43:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b24412640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c0b9900 con 0x7f7b1c0ced00 2024-03-18T14:43:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b1c15e020 0x7f7b1c162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b1c15e020 0x7f7b1c162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38240/0 (socket says 172.21.15.6:38240) 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 -- 172.21.15.6:0/1922558225 learned_addr learned my addr 172.21.15.6:0/1922558225 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b219ad640 1 --2- 172.21.15.6:0/1922558225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0ced00 0x7f7b1c0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b229af640 1 --2- 172.21.15.6:0/1922558225 >> v2:172.21.15.92:3300/0 conn(0x7f7b1c0d3210 0x7f7b1c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 -- 172.21.15.6:0/1922558225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0ced00 msgr2=0x7f7b1c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 --2- 172.21.15.6:0/1922558225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0ced00 0x7f7b1c0d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 -- 172.21.15.6:0/1922558225 >> v2:172.21.15.92:3300/0 conn(0x7f7b1c0d3210 msgr2=0x7f7b1c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 --2- 172.21.15.6:0/1922558225 >> v2:172.21.15.92:3300/0 conn(0x7f7b1c0d3210 0x7f7b1c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.681+0000 7f7b221ae640 1 -- 172.21.15.6:0/1922558225 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c0b9be0 con 0x7f7b1c15e020 2024-03-18T14:43:19.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.682+0000 7f7b221ae640 1 --2- 172.21.15.6:0/1922558225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c15e020 0x7f7b1c162380 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0b3390 tx=0x7f7b1802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d777caaa96fd0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:19.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.682+0000 7f7b211ac640 1 -- 172.21.15.6:0/1922558225 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1802e070 con 0x7f7b1c15e020 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.682+0000 7f7b211ac640 1 -- 172.21.15.6:0/1922558225 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b18037070 con 0x7f7b1c15e020 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.682+0000 7f7b211ac640 1 -- 172.21.15.6:0/1922558225 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18031a00 con 0x7f7b1c15e020 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 -- 172.21.15.6:0/1922558225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c15e020 msgr2=0x7f7b1c162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 --2- 172.21.15.6:0/1922558225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c15e020 0x7f7b1c162380 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0b3390 tx=0x7f7b1802b880 comp rx=0 tx=0).stop 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 -- 172.21.15.6:0/1922558225 shutdown_connections 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 --2- 172.21.15.6:0/1922558225 >> v2:172.21.15.92:3300/0 conn(0x7f7b1c0d3210 0x7f7b1c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 --2- 172.21.15.6:0/1922558225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0ced00 0x7f7b1c0d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 --2- 172.21.15.6:0/1922558225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c15e020 0x7f7b1c162380 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 -- 172.21.15.6:0/1922558225 >> 172.21.15.6:0/1922558225 conn(0x7f7b1c072ce0 msgr2=0x7f7b1c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 -- 172.21.15.6:0/1922558225 shutdown_connections 2024-03-18T14:43:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.683+0000 7f7b24412640 1 -- 172.21.15.6:0/1922558225 wait complete. 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.684+0000 7f7b24412640 1 Processor -- start 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.684+0000 7f7b24412640 1 -- start start 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b24412640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c0ced00 0x7f7b1c151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b24412640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0d3210 0x7f7b1c0b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b24412640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b1c15e020 0x7f7b1c15e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b24412640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c0cf440 con 0x7f7b1c0ced00 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b24412640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c0cf2c0 con 0x7f7b1c15e020 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b24412640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c0d20a0 con 0x7f7b1c0d3210 2024-03-18T14:43:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c0ced00 0x7f7b1c151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c0ced00 0x7f7b1c151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38254/0 (socket says 172.21.15.6:38254) 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 -- 172.21.15.6:0/3234549007 learned_addr learned my addr 172.21.15.6:0/3234549007 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b219ad640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0d3210 0x7f7b1c0b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b229af640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b1c15e020 0x7f7b1c15e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 -- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0d3210 msgr2=0x7f7b1c0b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0d3210 0x7f7b1c0b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 -- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b1c15e020 msgr2=0x7f7b1c15e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b1c15e020 0x7f7b1c15e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 -- 172.21.15.6:0/3234549007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c15eea0 con 0x7f7b1c0ced00 2024-03-18T14:43:19.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b219ad640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0d3210 0x7f7b1c0b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.685+0000 7f7b221ae640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c0ced00 0x7f7b1c151aa0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7b18000c00 tx=0x7f7b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.686+0000 7f7b12ffd640 1 -- 172.21.15.6:0/3234549007 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18032880 con 0x7f7b1c0ced00 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.686+0000 7f7b12ffd640 1 -- 172.21.15.6:0/3234549007 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b18032a40 con 0x7f7b1c0ced00 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.686+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1c15f180 con 0x7f7b1c0ced00 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.686+0000 7f7b12ffd640 1 -- 172.21.15.6:0/3234549007 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18032e30 con 0x7f7b1c0ced00 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.686+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1c0b4ba0 con 0x7f7b1c0ced00 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.687+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b1c08afe0 con 0x7f7b1c0ced00 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.687+0000 7f7b12ffd640 1 -- 172.21.15.6:0/3234549007 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7b180453f0 con 0x7f7b1c0ced00 2024-03-18T14:43:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.687+0000 7f7b12ffd640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b0006bbd0 0x7f7b0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.688+0000 7f7b12ffd640 1 -- 172.21.15.6:0/3234549007 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f7b180acdc0 con 0x7f7b1c0ced00 2024-03-18T14:43:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.688+0000 7f7b219ad640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b0006bbd0 0x7f7b0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.688+0000 7f7b219ad640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b0006bbd0 0x7f7b0006e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c15f500 tx=0x7f7b0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:19.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.692+0000 7f7b12ffd640 1 -- 172.21.15.6:0/3234549007 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7b18040070 con 0x7f7b1c0ced00 2024-03-18T14:43:19.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.873+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b1c0d1940 con 0x7f7b1c0ced00 2024-03-18T14:43:19.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.875+0000 7f7b12ffd640 1 -- 172.21.15.6:0/3234549007 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+18707 (secure 0 0 0) 0x7f7b1807b580 con 0x7f7b1c0ced00 2024-03-18T14:43:19.877 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:19.877 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:17.388447+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b0006bbd0 msgr2=0x7f7b0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b0006bbd0 0x7f7b0006e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c15f500 tx=0x7f7b0c008040 comp rx=0 tx=0).stop 2024-03-18T14:43:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c0ced00 msgr2=0x7f7b1c151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c0ced00 0x7f7b1c151aa0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7b18000c00 tx=0x7f7b180046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 shutdown_connections 2024-03-18T14:43:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7b0006bbd0 0x7f7b0006e090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b1c15e020 0x7f7b1c15e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b1c0d3210 0x7f7b1c0b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 --2- 172.21.15.6:0/3234549007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b1c0ced00 0x7f7b1c151aa0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 >> 172.21.15.6:0/3234549007 conn(0x7f7b1c072ce0 msgr2=0x7f7b1c15c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 shutdown_connections 2024-03-18T14:43:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.877+0000 7f7b24412640 1 -- 172.21.15.6:0/3234549007 wait complete. 2024-03-18T14:43:19.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:42:54.939648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:42:55.638120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-18T14:43:03.985735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:19.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:19.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.994+0000 7fa4af6a1640 1 Processor -- start 2024-03-18T14:43:19.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.994+0000 7fa4af6a1640 1 -- start start 2024-03-18T14:43:19.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4af6a1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4a80d0b60 0x7fa4a80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:19.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4af6a1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a80d39d0 con 0x7fa4a815e030 2024-03-18T14:43:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4af6a1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a80d2fd0 con 0x7fa4a80d3570 2024-03-18T14:43:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4af6a1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa4a815e410 con 0x7fa4a80d0b60 2024-03-18T14:43:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4adc3e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4a80d0b60 0x7fa4a80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4ad43d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa4a815e030 0x7fa4a80ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56260/0 (socket says 172.21.15.6:56260) 2024-03-18T14:43:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4ad43d640 1 -- 172.21.15.6:0/4046486802 learned_addr learned my addr 172.21.15.6:0/4046486802 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4adc3e640 1 -- 172.21.15.6:0/4046486802 >> v1:172.21.15.92:6789/0 conn(0x7fa4a80d3570 legacy=0x7fa4a80d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4adc3e640 1 -- 172.21.15.6:0/4046486802 >> v1:172.21.15.6:6789/0 conn(0x7fa4a815e030 legacy=0x7fa4a80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa49ffff640 1 -- 172.21.15.6:0/4046486802 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3139214308 0 0) 0x7fa4a80d39d0 con 0x7fa4a815e030 2024-03-18T14:43:20.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.995+0000 7fa4adc3e640 1 -- 172.21.15.6:0/4046486802 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a80d3290 con 0x7fa4a80d0b60 2024-03-18T14:43:20.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.996+0000 7fa4adc3e640 1 --2- 172.21.15.6:0/4046486802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a80b6140 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009d90 tx=0x7fa4a002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1618c3af0f82151 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.996+0000 7fa49ffff640 1 -- 172.21.15.6:0/4046486802 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0039070 con 0x7fa4a80d0b60 2024-03-18T14:43:20.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.996+0000 7fa49ffff640 1 -- 172.21.15.6:0/4046486802 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a002cd00 con 0x7fa4a80d0b60 2024-03-18T14:43:20.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.996+0000 7fa49ffff640 1 -- 172.21.15.6:0/4046486802 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0034660 con 0x7fa4a80d0b60 2024-03-18T14:43:20.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.996+0000 7fa4af6a1640 1 -- 172.21.15.6:0/4046486802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 msgr2=0x7fa4a80b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.996+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/4046486802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a80b6140 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009d90 tx=0x7fa4a002bf50 comp rx=0 tx=0).stop 2024-03-18T14:43:20.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.997+0000 7fa4af6a1640 1 -- 172.21.15.6:0/4046486802 shutdown_connections 2024-03-18T14:43:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.997+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/4046486802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a80b6140 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.997+0000 7fa4af6a1640 1 -- 172.21.15.6:0/4046486802 >> 172.21.15.6:0/4046486802 conn(0x7fa4a806c8c0 msgr2=0x7fa4a806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.997+0000 7fa4af6a1640 1 -- 172.21.15.6:0/4046486802 shutdown_connections 2024-03-18T14:43:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.997+0000 7fa4af6a1640 1 -- 172.21.15.6:0/4046486802 wait complete. 2024-03-18T14:43:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.998+0000 7fa4af6a1640 1 Processor -- start 2024-03-18T14:43:20.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.998+0000 7fa4af6a1640 1 -- start start 2024-03-18T14:43:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4af6a1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a8154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4af6a1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4a80d3570 0x7fa4a815f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4af6a1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4a815e030 0x7fa4a80cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4af6a1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a80ce830 con 0x7fa4a80d0b60 2024-03-18T14:43:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4af6a1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a80ce300 con 0x7fa4a80d3570 2024-03-18T14:43:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4af6a1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a8161a90 con 0x7fa4a815e030 2024-03-18T14:43:20.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a8154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a8154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38274/0 (socket says 172.21.15.6:38274) 2024-03-18T14:43:20.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 -- 172.21.15.6:0/1012540872 learned_addr learned my addr 172.21.15.6:0/1012540872 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:20.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4acc3c640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4a80d3570 0x7fa4a815f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4adc3e640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4a815e030 0x7fa4a80cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 -- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4a815e030 msgr2=0x7fa4a80cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4a815e030 0x7fa4a80cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 -- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4a80d3570 msgr2=0x7fa4a815f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4a80d3570 0x7fa4a815f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 -- 172.21.15.6:0/1012540872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a815fe90 con 0x7fa4a80d0b60 2024-03-18T14:43:20.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4acc3c640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4a80d3570 0x7fa4a815f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:20.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4adc3e640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4a815e030 0x7fa4a80cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:20.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4ad43d640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a8154a50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c3f0 tx=0x7fa4a400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa49dffb640 1 -- 172.21.15.6:0/1012540872 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40074e0 con 0x7fa4a80d0b60 2024-03-18T14:43:20.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa49dffb640 1 -- 172.21.15.6:0/1012540872 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a40047e0 con 0x7fa4a80d0b60 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a8160170 con 0x7fa4a80d0b60 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:19.999+0000 7fa49dffb640 1 -- 172.21.15.6:0/1012540872 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4004bd0 con 0x7fa4a80d0b60 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.000+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a8160730 con 0x7fa4a80d0b60 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.000+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a8160b70 con 0x7fa4a80d0b60 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.001+0000 7fa49dffb640 1 -- 172.21.15.6:0/1012540872 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa4a4011070 con 0x7fa4a80d0b60 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.001+0000 7fa49dffb640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa48c06bbd0 0x7fa48c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.001+0000 7fa49dffb640 1 -- 172.21.15.6:0/1012540872 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7fa4a408a9e0 con 0x7fa4a80d0b60 2024-03-18T14:43:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.001+0000 7fa4acc3c640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa48c06bbd0 0x7fa48c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.002+0000 7fa4acc3c640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa48c06bbd0 0x7fa48c06e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa4a81559e0 tx=0x7fa498009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.006+0000 7fa49dffb640 1 -- 172.21.15.6:0/1012540872 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa4a40591a0 con 0x7fa4a80d0b60 2024-03-18T14:43:20.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.186+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4a80d19f0 con 0x7fa4a80d0b60 2024-03-18T14:43:20.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.186+0000 7fa49dffb640 1 -- 172.21.15.6:0/1012540872 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7fa4a4058aa0 con 0x7fa4a80d0b60 2024-03-18T14:43:20.188 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:20.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa48c06bbd0 msgr2=0x7fa48c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa48c06bbd0 0x7fa48c06e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa4a81559e0 tx=0x7fa498009040 comp rx=0 tx=0).stop 2024-03-18T14:43:20.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 msgr2=0x7fa4a8154a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a8154a50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c3f0 tx=0x7fa4a400c8b0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 shutdown_connections 2024-03-18T14:43:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa48c06bbd0 0x7fa48c06e090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4a815e030 0x7fa4a80cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4a80d3570 0x7fa4a815f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 --2- 172.21.15.6:0/1012540872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4a80d0b60 0x7fa4a8154a50 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 >> 172.21.15.6:0/1012540872 conn(0x7fa4a806c8c0 msgr2=0x7fa4a80b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 shutdown_connections 2024-03-18T14:43:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.188+0000 7fa4af6a1640 1 -- 172.21.15.6:0/1012540872 wait complete. 2024-03-18T14:43:20.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:20.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.307+0000 7f2b223d4640 1 Processor -- start 2024-03-18T14:43:20.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.307+0000 7f2b223d4640 1 -- start start 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b223d4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2b1c15e000 0x7f2b1c0b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b223d4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c067330 0x7f2b1c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b223d4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c0ad080 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b223d4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2b1c0ad360 con 0x7f2b1c15e000 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b223d4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c0ad640 con 0x7f2b1c067330 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c067330 0x7f2b1c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1bfff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2b1c15e000 0x7f2b1c0b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c067330 0x7f2b1c0b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45870/0 (socket says 172.21.15.6:45870) 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b20971640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2b1c0ae3a0 0x7f2b1c0721f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:33808/0 (socket says 172.21.15.6:33808) 2024-03-18T14:43:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3517158851 learned_addr learned my addr 172.21.15.6:0/3517158851 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1affd640 1 -- 172.21.15.6:0/3517158851 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975136796 0 0) 0x7f2b1c0ad080 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1affd640 1 -- 172.21.15.6:0/3517158851 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b08003520 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3517158851 >> v2:172.21.15.92:3300/0 conn(0x7f2b1c15e000 msgr2=0x7f2b1c0b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1b7fe640 1 --2- 172.21.15.6:0/3517158851 >> v2:172.21.15.92:3300/0 conn(0x7f2b1c15e000 0x7f2b1c0b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.308+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3517158851 >> v1:172.21.15.92:6789/0 conn(0x7f2b1c0ae3a0 legacy=0x7f2b1c0721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.309+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3517158851 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c0ad920 con 0x7f2b1c067330 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.309+0000 7f2b1bfff640 1 --2- 172.21.15.6:0/3517158851 >> v2:172.21.15.92:3300/0 conn(0x7f2b1c15e000 0x7f2b1c0b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.309+0000 7f2b1b7fe640 1 --2- 172.21.15.6:0/3517158851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c067330 0x7f2b1c0b2170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009870 tx=0x7f2b0402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99615fd92a35f43 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b1affd640 1 -- 172.21.15.6:0/3517158851 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04034070 con 0x7f2b1c067330 2024-03-18T14:43:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b1affd640 1 -- 172.21.15.6:0/3517158851 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b04030840 con 0x7f2b1c067330 2024-03-18T14:43:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b1affd640 1 -- 172.21.15.6:0/3517158851 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04030c10 con 0x7f2b1c067330 2024-03-18T14:43:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 -- 172.21.15.6:0/3517158851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c067330 msgr2=0x7f2b1c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3517158851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c067330 0x7f2b1c0b2170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009870 tx=0x7f2b0402b980 comp rx=0 tx=0).stop 2024-03-18T14:43:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 -- 172.21.15.6:0/3517158851 shutdown_connections 2024-03-18T14:43:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3517158851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c067330 0x7f2b1c0b2170 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3517158851 >> v2:172.21.15.92:3300/0 conn(0x7f2b1c15e000 0x7f2b1c0b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 -- 172.21.15.6:0/3517158851 >> 172.21.15.6:0/3517158851 conn(0x7f2b1c0ccc30 msgr2=0x7f2b1c0cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 -- 172.21.15.6:0/3517158851 shutdown_connections 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.310+0000 7f2b223d4640 1 -- 172.21.15.6:0/3517158851 wait complete. 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.311+0000 7f2b223d4640 1 Processor -- start 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b223d4640 1 -- start start 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b223d4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b1c067330 0x7f2b1c154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b223d4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b1c0ae3a0 0x7f2b1c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b223d4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c15e000 0x7f2b1c159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b223d4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0cd8a0 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b223d4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0cd720 con 0x7f2b1c067330 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b223d4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c15f3f0 con 0x7f2b1c15e000 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.312+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b1c0ae3a0 0x7f2b1c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b1c0ae3a0 0x7f2b1c159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38280/0 (socket says 172.21.15.6:38280) 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3289366435 learned_addr learned my addr 172.21.15.6:0/3289366435 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b20971640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c15e000 0x7f2b1c159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1bfff640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b1c067330 0x7f2b1c154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c15e000 msgr2=0x7f2b1c159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c15e000 0x7f2b1c159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b1c067330 msgr2=0x7f2b1c154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b1c067330 0x7f2b1c154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 -- 172.21.15.6:0/3289366435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c0c6940 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b1b7fe640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b1c0ae3a0 0x7f2b1c159a90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2b0402b9c0 tx=0x7f2b04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b18ff9640 1 -- 172.21.15.6:0/3289366435 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04048070 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c15a600 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b18ff9640 1 -- 172.21.15.6:0/3289366435 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b04039a90 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b18ff9640 1 -- 172.21.15.6:0/3289366435 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04034040 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.313+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c15abc0 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.315+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c068420 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.315+0000 7f2b18ff9640 1 -- 172.21.15.6:0/3289366435 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2b04037070 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.315+0000 7f2b18ff9640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2afc06bbd0 0x7f2afc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.315+0000 7f2b18ff9640 1 -- 172.21.15.6:0/3289366435 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f2b040ac9a0 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.316+0000 7f2b1bfff640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2afc06bbd0 0x7f2afc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.316+0000 7f2b1bfff640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2afc06bbd0 0x7f2afc06e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009a30 tx=0x7f2b0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.321+0000 7f2b18ff9640 1 -- 172.21.15.6:0/3289366435 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2b0407b0e0 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.583+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2b1c068210 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.584+0000 7f2b18ff9640 1 -- 172.21.15.6:0/3289366435 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7f2b1c068210 con 0x7f2b1c0ae3a0 2024-03-18T14:43:20.585 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:20.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2afc06bbd0 msgr2=0x7f2afc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2afc06bbd0 0x7f2afc06e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009a30 tx=0x7f2b0c008040 comp rx=0 tx=0).stop 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b1c0ae3a0 msgr2=0x7f2b1c159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b1c0ae3a0 0x7f2b1c159a90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2b0402b9c0 tx=0x7f2b04004730 comp rx=0 tx=0).stop 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 shutdown_connections 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2afc06bbd0 0x7f2afc06e090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2b1c15e000 0x7f2b1c159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b1c0ae3a0 0x7f2b1c159a90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 --2- 172.21.15.6:0/3289366435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b1c067330 0x7f2b1c154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 >> 172.21.15.6:0/3289366435 conn(0x7f2b1c0ccc30 msgr2=0x7f2b1c0d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 shutdown_connections 2024-03-18T14:43:20.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.586+0000 7f2b223d4640 1 -- 172.21.15.6:0/3289366435 wait complete. 2024-03-18T14:43:20.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:20.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.701+0000 7f2e0c374640 1 Processor -- start 2024-03-18T14:43:20.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.701+0000 7f2e0c374640 1 -- start start 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0c374640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e0415dfd0 0x7f2e040b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0c374640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2e040b5740 0x7f2e040b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0c374640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e0415e3b0 con 0x7f2e040d1e40 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0c374640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f2e040b8470 con 0x7f2e0415dfd0 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0c374640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f2e040b8750 con 0x7f2e040b5740 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0a110640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e0415dfd0 0x7f2e040b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0a110640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e0415dfd0 0x7f2e040b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38282/0 (socket says 172.21.15.6:38282) 2024-03-18T14:43:20.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0a110640 1 -- 172.21.15.6:0/121764812 learned_addr learned my addr 172.21.15.6:0/121764812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:20.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0990f640 1 --2- 172.21.15.6:0/121764812 >> v2:172.21.15.92:3300/0 conn(0x7f2e040b5740 0x7f2e040b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0a110640 1 -- 172.21.15.6:0/121764812 >> v2:172.21.15.92:3300/0 conn(0x7f2e040b5740 msgr2=0x7f2e040b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:20.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0a110640 1 --2- 172.21.15.6:0/121764812 >> v2:172.21.15.92:3300/0 conn(0x7f2e040b5740 0x7f2e040b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0a110640 1 -- 172.21.15.6:0/121764812 >> v1:172.21.15.6:6789/0 conn(0x7f2e040d1e40 legacy=0x7f2e040d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0a110640 1 -- 172.21.15.6:0/121764812 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e040b8a30 con 0x7f2e0415dfd0 2024-03-18T14:43:20.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0910e640 1 -- 172.21.15.6:0/121764812 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068965403 0 0) 0x7f2e0415e3b0 con 0x7f2e040d1e40 2024-03-18T14:43:20.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.702+0000 7f2e0990f640 1 --2- 172.21.15.6:0/121764812 >> v2:172.21.15.92:3300/0 conn(0x7f2e040b5740 0x7f2e040b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.703+0000 7f2e0a110640 1 --2- 172.21.15.6:0/121764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0415dfd0 0x7f2e040b9650 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2e040ce4a0 tx=0x7f2df802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f918a1cf9f844a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.703+0000 7f2e0910e640 1 -- 172.21.15.6:0/121764812 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df8034070 con 0x7f2e0415dfd0 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.703+0000 7f2e0910e640 1 -- 172.21.15.6:0/121764812 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2df8030860 con 0x7f2e0415dfd0 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.703+0000 7f2e0910e640 1 -- 172.21.15.6:0/121764812 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df8030c30 con 0x7f2e0415dfd0 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 -- 172.21.15.6:0/121764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0415dfd0 msgr2=0x7f2e040b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 --2- 172.21.15.6:0/121764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0415dfd0 0x7f2e040b9650 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2e040ce4a0 tx=0x7f2df802b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 -- 172.21.15.6:0/121764812 shutdown_connections 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 --2- 172.21.15.6:0/121764812 >> v2:172.21.15.92:3300/0 conn(0x7f2e040b5740 0x7f2e040b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 --2- 172.21.15.6:0/121764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e0415dfd0 0x7f2e040b9650 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 -- 172.21.15.6:0/121764812 >> 172.21.15.6:0/121764812 conn(0x7f2e04072d10 msgr2=0x7f2e04073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:20.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 -- 172.21.15.6:0/121764812 shutdown_connections 2024-03-18T14:43:20.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.704+0000 7f2e0c374640 1 -- 172.21.15.6:0/121764812 wait complete. 2024-03-18T14:43:20.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.706+0000 7f2e0c374640 1 Processor -- start 2024-03-18T14:43:20.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.706+0000 7f2e0c374640 1 -- start start 2024-03-18T14:43:20.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0c374640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e040b5740 0x7f2e041f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0c374640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040d1e40 0x7f2e0415fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0c374640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0415dfd0 0x7f2e040d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0c374640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2e040b7e00 con 0x7f2e040d1e40 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0c374640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2e040b9b90 con 0x7f2e0415dfd0 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0c374640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2e040b7f80 con 0x7f2e040b5740 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0990f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040d1e40 0x7f2e0415fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0990f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040d1e40 0x7f2e0415fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38284/0 (socket says 172.21.15.6:38284) 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0990f640 1 -- 172.21.15.6:0/2631206574 learned_addr learned my addr 172.21.15.6:0/2631206574 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0a110640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e040b5740 0x7f2e041f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.707+0000 7f2e0a911640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0415dfd0 0x7f2e040d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0990f640 1 -- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e040b5740 msgr2=0x7f2e041f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0990f640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e040b5740 0x7f2e041f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0990f640 1 -- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0415dfd0 msgr2=0x7f2e040d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0990f640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0415dfd0 0x7f2e040d2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0990f640 1 -- 172.21.15.6:0/2631206574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e04160260 con 0x7f2e040d1e40 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0990f640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040d1e40 0x7f2e0415fd20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c7f0 tx=0x7f2e0000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2df6ffd640 1 -- 172.21.15.6:0/2631206574 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00015070 con 0x7f2e040d1e40 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2df6ffd640 1 -- 172.21.15.6:0/2631206574 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e00007a30 con 0x7f2e040d1e40 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e041f74d0 con 0x7f2e040d1e40 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2df6ffd640 1 -- 172.21.15.6:0/2631206574 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00007e00 con 0x7f2e040d1e40 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.708+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e041f7a90 con 0x7f2e040d1e40 2024-03-18T14:43:20.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.709+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e040751b0 con 0x7f2e040d1e40 2024-03-18T14:43:20.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.710+0000 7f2df6ffd640 1 -- 172.21.15.6:0/2631206574 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2e00018070 con 0x7f2e040d1e40 2024-03-18T14:43:20.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.710+0000 7f2df6ffd640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2de806bbd0 0x7f2de806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:20.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.710+0000 7f2df6ffd640 1 -- 172.21.15.6:0/2631206574 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f2e0008ac10 con 0x7f2e040d1e40 2024-03-18T14:43:20.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.710+0000 7f2e0a110640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2de806bbd0 0x7f2de806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:20.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.711+0000 7f2e0a110640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2de806bbd0 0x7f2de806e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2df8036000 tx=0x7f2df803b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:20.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.714+0000 7f2df6ffd640 1 -- 172.21.15.6:0/2631206574 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2e00058e10 con 0x7f2e040d1e40 2024-03-18T14:43:20.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.898+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e040b6300 con 0x7f2e040d1e40 2024-03-18T14:43:20.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.899+0000 7f2df6ffd640 1 -- 172.21.15.6:0/2631206574 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7f2e0005ceb0 con 0x7f2e040d1e40 2024-03-18T14:43:20.900 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2de806bbd0 msgr2=0x7f2de806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2de806bbd0 0x7f2de806e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2df8036000 tx=0x7f2df803b040 comp rx=0 tx=0).stop 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040d1e40 msgr2=0x7f2e0415fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040d1e40 0x7f2e0415fd20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c7f0 tx=0x7f2e0000cbf0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 shutdown_connections 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2de806bbd0 0x7f2de806e090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0415dfd0 0x7f2e040d2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040d1e40 0x7f2e0415fd20 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 --2- 172.21.15.6:0/2631206574 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e040b5740 0x7f2e041f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 >> 172.21.15.6:0/2631206574 conn(0x7f2e04072d10 msgr2=0x7f2e040ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 shutdown_connections 2024-03-18T14:43:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:20.901+0000 7f2e0c374640 1 -- 172.21.15.6:0/2631206574 wait complete. 2024-03-18T14:43:20.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-18T14:43:21.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.016+0000 7f8cacf05640 1 Processor -- start 2024-03-18T14:43:21.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.016+0000 7f8cacf05640 1 -- start start 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8cacf05640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ca806d1f0 0x7f8ca80d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8cacf05640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8ca80d0680 0x7f8ca815d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8cacf05640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ca80d0f30 con 0x7f8ca80ce930 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8cacf05640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8ca80b1070 con 0x7f8ca806d1f0 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8cacf05640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f8ca80b1350 con 0x7f8ca80d0680 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8ca5d9b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ca806d1f0 0x7f8ca80d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8ca5d9b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ca806d1f0 0x7f8ca80d0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38300/0 (socket says 172.21.15.6:38300) 2024-03-18T14:43:21.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.017+0000 7f8ca5d9b640 1 -- 172.21.15.6:0/248462190 learned_addr learned my addr 172.21.15.6:0/248462190 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:21.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca5d9b640 1 -- 172.21.15.6:0/248462190 >> v2:172.21.15.92:3300/0 conn(0x7f8ca80d0680 msgr2=0x7f8ca815d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:21.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca5d9b640 1 --2- 172.21.15.6:0/248462190 >> v2:172.21.15.92:3300/0 conn(0x7f8ca80d0680 0x7f8ca815d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca5d9b640 1 -- 172.21.15.6:0/248462190 >> v1:172.21.15.92:6789/0 conn(0x7f8ca80ce930 legacy=0x7f8ca806cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:21.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca5d9b640 1 -- 172.21.15.6:0/248462190 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca80ced10 con 0x7f8ca806d1f0 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca5d9b640 1 --2- 172.21.15.6:0/248462190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca806d1f0 0x7f8ca80d0140 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80c5520 tx=0x7f8c9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64361258a306461e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca559a640 1 -- 172.21.15.6:0/248462190 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90035020 con 0x7f8ca806d1f0 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca559a640 1 -- 172.21.15.6:0/248462190 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c90031870 con 0x7f8ca806d1f0 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.018+0000 7f8ca559a640 1 -- 172.21.15.6:0/248462190 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90031c40 con 0x7f8ca806d1f0 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 -- 172.21.15.6:0/248462190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca806d1f0 msgr2=0x7f8ca80d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 --2- 172.21.15.6:0/248462190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca806d1f0 0x7f8ca80d0140 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80c5520 tx=0x7f8c9002b910 comp rx=0 tx=0).stop 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 -- 172.21.15.6:0/248462190 shutdown_connections 2024-03-18T14:43:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 --2- 172.21.15.6:0/248462190 >> v2:172.21.15.92:3300/0 conn(0x7f8ca80d0680 0x7f8ca815d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 --2- 172.21.15.6:0/248462190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca806d1f0 0x7f8ca80d0140 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 -- 172.21.15.6:0/248462190 >> 172.21.15.6:0/248462190 conn(0x7f8ca808b0c0 msgr2=0x7f8ca80d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 -- 172.21.15.6:0/248462190 shutdown_connections 2024-03-18T14:43:21.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.019+0000 7f8cacf05640 1 -- 172.21.15.6:0/248462190 wait complete. 2024-03-18T14:43:21.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.020+0000 7f8cacf05640 1 Processor -- start 2024-03-18T14:43:21.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.020+0000 7f8cacf05640 1 -- start start 2024-03-18T14:43:21.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8cacf05640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ca806d1f0 0x7f8ca81499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8cacf05640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ca80ce930 0x7f8ca80c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8cacf05640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca80d0680 0x7f8ca80b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8cacf05640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca815dbf0 con 0x7f8ca80d0680 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8cacf05640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca80d0ce0 con 0x7f8ca806d1f0 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8cacf05640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca815dd70 con 0x7f8ca80ce930 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8ca6d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca80d0680 0x7f8ca80b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8ca6d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca80d0680 0x7f8ca80b6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38314/0 (socket says 172.21.15.6:38314) 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.021+0000 7f8ca6d9d640 1 -- 172.21.15.6:0/3997363864 learned_addr learned my addr 172.21.15.6:0/3997363864 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca5d9b640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ca80ce930 0x7f8ca80c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca659c640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ca806d1f0 0x7f8ca81499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca6d9d640 1 -- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ca80ce930 msgr2=0x7f8ca80c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca6d9d640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ca80ce930 0x7f8ca80c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca6d9d640 1 -- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ca806d1f0 msgr2=0x7f8ca81499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca6d9d640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ca806d1f0 0x7f8ca81499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca6d9d640 1 -- 172.21.15.6:0/3997363864 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca80c5ca0 con 0x7f8ca80d0680 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8ca6d9d640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca80d0680 0x7f8ca80b6c70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000b970 tx=0x7f8ca000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8c9effd640 1 -- 172.21.15.6:0/3997363864 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0019070 con 0x7f8ca80d0680 2024-03-18T14:43:21.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8c9effd640 1 -- 172.21.15.6:0/3997363864 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ca000cb60 con 0x7f8ca80d0680 2024-03-18T14:43:21.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca80c5f80 con 0x7f8ca80d0680 2024-03-18T14:43:21.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8c9effd640 1 -- 172.21.15.6:0/3997363864 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0014440 con 0x7f8ca80d0680 2024-03-18T14:43:21.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.022+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca80c6540 con 0x7f8ca80d0680 2024-03-18T14:43:21.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.023+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca80751b0 con 0x7f8ca80d0680 2024-03-18T14:43:21.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.024+0000 7f8c9effd640 1 -- 172.21.15.6:0/3997363864 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8ca001c070 con 0x7f8ca80d0680 2024-03-18T14:43:21.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.024+0000 7f8c9effd640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c8806bbd0 0x7f8c8806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.024+0000 7f8c9effd640 1 -- 172.21.15.6:0/3997363864 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f8ca008aae0 con 0x7f8ca80d0680 2024-03-18T14:43:21.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.024+0000 7f8ca659c640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c8806bbd0 0x7f8c8806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.025+0000 7f8ca659c640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c8806bbd0 0x7f8c8806e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8c98004e50 tx=0x7f8c98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.028+0000 7f8c9effd640 1 -- 172.21.15.6:0/3997363864 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8ca0059330 con 0x7f8ca80d0680 2024-03-18T14:43:21.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.210+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f8ca808b680 con 0x7f8ca80d0680 2024-03-18T14:43:21.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.211+0000 7f8c9effd640 1 -- 172.21.15.6:0/3997363864 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v44) ==== 97+0+11 (secure 0 0 0) 0x7f8ca0058c30 con 0x7f8ca80d0680 2024-03-18T14:43:21.212 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:21.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c8806bbd0 msgr2=0x7f8c8806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c8806bbd0 0x7f8c8806e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8c98004e50 tx=0x7f8c98009040 comp rx=0 tx=0).stop 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca80d0680 msgr2=0x7f8ca80b6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca80d0680 0x7f8ca80b6c70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000b970 tx=0x7f8ca000be30 comp rx=0 tx=0).stop 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 shutdown_connections 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c8806bbd0 0x7f8c8806e090 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ca80d0680 0x7f8ca80b6c70 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8ca80ce930 0x7f8ca80c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 --2- 172.21.15.6:0/3997363864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ca806d1f0 0x7f8ca81499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 >> 172.21.15.6:0/3997363864 conn(0x7f8ca808b0c0 msgr2=0x7f8ca806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 shutdown_connections 2024-03-18T14:43:21.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.213+0000 7f8cacf05640 1 -- 172.21.15.6:0/3997363864 wait complete. 2024-03-18T14:43:21.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.328+0000 7f02be310640 1 Processor -- start 2024-03-18T14:43:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.328+0000 7f02be310640 1 -- start start 2024-03-18T14:43:21.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02be310640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f02b80d2fd0 0x7f02b80b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02be310640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b80d3430 con 0x7f02b80b6df0 2024-03-18T14:43:21.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02be310640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b80d1f30 con 0x7f02b806c7a0 2024-03-18T14:43:21.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02be310640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f02b80d2210 con 0x7f02b80d2fd0 2024-03-18T14:43:21.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02bc8ad640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f02b80b6df0 0x7f02b80b7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56280/0 (socket says 172.21.15.6:56280) 2024-03-18T14:43:21.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02bc8ad640 1 -- 172.21.15.6:0/2418758018 learned_addr learned my addr 172.21.15.6:0/2418758018 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02b77fe640 1 --2- 172.21.15.6:0/2418758018 >> v2:172.21.15.92:3300/0 conn(0x7f02b80d2fd0 0x7f02b80b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02b6ffd640 1 -- 172.21.15.6:0/2418758018 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3281072893 0 0) 0x7f02b80d3430 con 0x7f02b80b6df0 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.329+0000 7f02b6ffd640 1 -- 172.21.15.6:0/2418758018 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02a0003520 con 0x7f02b80b6df0 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.330+0000 7f02b6ffd640 1 -- 172.21.15.6:0/2418758018 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656690752 0 0) 0x7f02b80d1f30 con 0x7f02b806c7a0 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.330+0000 7f02b6ffd640 1 -- 172.21.15.6:0/2418758018 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b80d3430 con 0x7f02b806c7a0 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.330+0000 7f02b77fe640 1 -- 172.21.15.6:0/2418758018 >> v1:172.21.15.92:6789/0 conn(0x7f02b806c7a0 legacy=0x7f02b806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.330+0000 7f02b77fe640 1 -- 172.21.15.6:0/2418758018 >> v1:172.21.15.6:6789/0 conn(0x7f02b80b6df0 legacy=0x7f02b80b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.330+0000 7f02b77fe640 1 -- 172.21.15.6:0/2418758018 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b80d24f0 con 0x7f02b80d2fd0 2024-03-18T14:43:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.330+0000 7f02b77fe640 1 --2- 172.21.15.6:0/2418758018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b80b95e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0097a0 tx=0x7f02ac02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df5bd0790c3be539 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02b6ffd640 1 -- 172.21.15.6:0/2418758018 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac02f810 con 0x7f02b80d2fd0 2024-03-18T14:43:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02b6ffd640 1 -- 172.21.15.6:0/2418758018 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02ac02f9b0 con 0x7f02b80d2fd0 2024-03-18T14:43:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02b6ffd640 1 -- 172.21.15.6:0/2418758018 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac02fd80 con 0x7f02b80d2fd0 2024-03-18T14:43:21.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02be310640 1 -- 172.21.15.6:0/2418758018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 msgr2=0x7f02b80b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02be310640 1 --2- 172.21.15.6:0/2418758018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b80b95e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0097a0 tx=0x7f02ac02b9f0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02be310640 1 -- 172.21.15.6:0/2418758018 shutdown_connections 2024-03-18T14:43:21.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02be310640 1 --2- 172.21.15.6:0/2418758018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b80b95e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02be310640 1 -- 172.21.15.6:0/2418758018 >> 172.21.15.6:0/2418758018 conn(0x7f02b8072ce0 msgr2=0x7f02b80730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.331+0000 7f02be310640 1 -- 172.21.15.6:0/2418758018 shutdown_connections 2024-03-18T14:43:21.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.332+0000 7f02be310640 1 -- 172.21.15.6:0/2418758018 wait complete. 2024-03-18T14:43:21.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.333+0000 7f02be310640 1 Processor -- start 2024-03-18T14:43:21.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.333+0000 7f02be310640 1 -- start start 2024-03-18T14:43:21.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02be310640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b806c7a0 0x7f02b814c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02be310640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b80b6df0 0x7f02b81e8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02be310640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b81e9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02be310640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f02b80b9b20 con 0x7f02b806c7a0 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02be310640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02b80d2ab0 con 0x7f02b80d2fd0 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02be310640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f02b80b9ca0 con 0x7f02b80b6df0 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b806c7a0 0x7f02b814c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b806c7a0 0x7f02b814c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38330/0 (socket says 172.21.15.6:38330) 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 -- 172.21.15.6:0/2557763483 learned_addr learned my addr 172.21.15.6:0/2557763483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02bc8ad640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b81e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b77fe640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b80b6df0 0x7f02b81e8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 -- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b80b6df0 msgr2=0x7f02b81e8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b80b6df0 0x7f02b81e8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 -- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 msgr2=0x7f02b81e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b81e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 -- 172.21.15.6:0/2557763483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b80b5f40 con 0x7f02b806c7a0 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02bc8ad640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b81e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.334+0000 7f02b7fff640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b806c7a0 0x7f02b814c170 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f02a800c490 tx=0x7f02a800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.335+0000 7f02b4ff9640 1 -- 172.21.15.6:0/2557763483 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a80052d0 con 0x7f02b806c7a0 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.335+0000 7f02b4ff9640 1 -- 172.21.15.6:0/2557763483 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a8005470 con 0x7f02b806c7a0 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.335+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b81e65e0 con 0x7f02b806c7a0 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.335+0000 7f02b4ff9640 1 -- 172.21.15.6:0/2557763483 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a8012870 con 0x7f02b806c7a0 2024-03-18T14:43:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.335+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b80b5180 con 0x7f02b806c7a0 2024-03-18T14:43:21.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.336+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b80d37f0 con 0x7f02b806c7a0 2024-03-18T14:43:21.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.337+0000 7f02b4ff9640 1 -- 172.21.15.6:0/2557763483 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f02a8012a10 con 0x7f02b806c7a0 2024-03-18T14:43:21.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.337+0000 7f02b4ff9640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f02a406bbd0 0x7f02a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.337+0000 7f02b4ff9640 1 -- 172.21.15.6:0/2557763483 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f02a808a8f0 con 0x7f02b806c7a0 2024-03-18T14:43:21.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.337+0000 7f02b77fe640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f02a406bbd0 0x7f02a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.338+0000 7f02b77fe640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f02a406bbd0 0x7f02a406e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f02ac036000 tx=0x7f02ac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.343+0000 7f02b4ff9640 1 -- 172.21.15.6:0/2557763483 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f02b80d37f0 con 0x7f02b806c7a0 2024-03-18T14:43:21.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.545+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f02b804f5b0 con 0x7f02b806c7a0 2024-03-18T14:43:21.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.546+0000 7f02b4ff9640 1 -- 172.21.15.6:0/2557763483 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) ==== 80+0+0 (secure 0 0 0) 0x7f02a8058a60 con 0x7f02b806c7a0 2024-03-18T14:43:21.547 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f02a406bbd0 msgr2=0x7f02a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f02a406bbd0 0x7f02a406e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f02ac036000 tx=0x7f02ac008030 comp rx=0 tx=0).stop 2024-03-18T14:43:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b806c7a0 msgr2=0x7f02b814c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b806c7a0 0x7f02b814c170 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f02a800c490 tx=0x7f02a800c950 comp rx=0 tx=0).stop 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 shutdown_connections 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f02a406bbd0 0x7f02a406e090 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02b80d2fd0 0x7f02b81e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02b80b6df0 0x7f02b81e8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 --2- 172.21.15.6:0/2557763483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b806c7a0 0x7f02b814c170 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 >> 172.21.15.6:0/2557763483 conn(0x7f02b8072ce0 msgr2=0x7f02b80ce280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 shutdown_connections 2024-03-18T14:43:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.548+0000 7f02be310640 1 -- 172.21.15.6:0/2557763483 wait complete. 2024-03-18T14:43:21.562 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:21.562 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:21.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.717+0000 7f1ddad6f640 1 Processor -- start 2024-03-18T14:43:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.717+0000 7f1ddad6f640 1 -- start start 2024-03-18T14:43:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1ddad6f640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1dd415bd40 0x7f1dd40d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1ddad6f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1dd40cfea0 0x7f1dd40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1ddad6f640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dd40cc870 con 0x7f1dd40d1d90 2024-03-18T14:43:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1ddad6f640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1dd40d02d0 con 0x7f1dd40cfea0 2024-03-18T14:43:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1ddad6f640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1dd40d0500 con 0x7f1dd415bd40 2024-03-18T14:43:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd930c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1dd40cfea0 0x7f1dd40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd930c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1dd40cfea0 0x7f1dd40cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38340/0 (socket says 172.21.15.6:38340) 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd930c640 1 -- 172.21.15.6:0/2310906851 learned_addr learned my addr 172.21.15.6:0/2310906851 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd8b0b640 1 --2- 172.21.15.6:0/2310906851 >> v2:172.21.15.92:3300/0 conn(0x7f1dd415bd40 0x7f1dd40d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd930c640 1 -- 172.21.15.6:0/2310906851 >> v2:172.21.15.92:3300/0 conn(0x7f1dd415bd40 msgr2=0x7f1dd40d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd930c640 1 --2- 172.21.15.6:0/2310906851 >> v2:172.21.15.92:3300/0 conn(0x7f1dd415bd40 0x7f1dd40d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd930c640 1 -- 172.21.15.6:0/2310906851 >> v1:172.21.15.92:6789/0 conn(0x7f1dd40d1d90 legacy=0x7f1dd40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.718+0000 7f1dd930c640 1 -- 172.21.15.6:0/2310906851 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd40d07e0 con 0x7f1dd40cfea0 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.719+0000 7f1dd930c640 1 --2- 172.21.15.6:0/2310906851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40cfea0 0x7f1dd40cd030 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40cca40 tx=0x7f1dcc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff075810fa33ffa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.719+0000 7f1dd37fe640 1 -- 172.21.15.6:0/2310906851 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc02ca70 con 0x7f1dd40cfea0 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.719+0000 7f1dd37fe640 1 -- 172.21.15.6:0/2310906851 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dcc02cc10 con 0x7f1dd40cfea0 2024-03-18T14:43:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.719+0000 7f1dd37fe640 1 -- 172.21.15.6:0/2310906851 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc0343f0 con 0x7f1dd40cfea0 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 -- 172.21.15.6:0/2310906851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40cfea0 msgr2=0x7f1dd40cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/2310906851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40cfea0 0x7f1dd40cd030 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40cca40 tx=0x7f1dcc02bed0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 -- 172.21.15.6:0/2310906851 shutdown_connections 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/2310906851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40cfea0 0x7f1dd40cd030 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/2310906851 >> v2:172.21.15.92:3300/0 conn(0x7f1dd415bd40 0x7f1dd40d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 -- 172.21.15.6:0/2310906851 >> 172.21.15.6:0/2310906851 conn(0x7f1dd409ae20 msgr2=0x7f1dd408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 -- 172.21.15.6:0/2310906851 shutdown_connections 2024-03-18T14:43:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.720+0000 7f1ddad6f640 1 -- 172.21.15.6:0/2310906851 wait complete. 2024-03-18T14:43:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.721+0000 7f1ddad6f640 1 Processor -- start 2024-03-18T14:43:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.721+0000 7f1ddad6f640 1 -- start start 2024-03-18T14:43:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1ddad6f640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1dd40cfea0 0x7f1dd4155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1ddad6f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40d1d90 0x7f1dd406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1ddad6f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dd415bd40 0x7f1dd406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1ddad6f640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd40d0ac0 con 0x7f1dd40d1d90 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1ddad6f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd40d2210 con 0x7f1dd415bd40 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1ddad6f640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1dd40d0c40 con 0x7f1dd40cfea0 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40d1d90 0x7f1dd406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40d1d90 0x7f1dd406d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38348/0 (socket says 172.21.15.6:38348) 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 -- 172.21.15.6:0/3146888266 learned_addr learned my addr 172.21.15.6:0/3146888266 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd930c640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dd415bd40 0x7f1dd406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd8b0b640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1dd40cfea0 0x7f1dd4155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 -- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1dd40cfea0 msgr2=0x7f1dd4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1dd40cfea0 0x7f1dd4155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 -- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dd415bd40 msgr2=0x7f1dd406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dd415bd40 0x7f1dd406da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 -- 172.21.15.6:0/3146888266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd406e0d0 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd3fff640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40d1d90 0x7f1dd406d510 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000c460 tx=0x7f1dc000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.722+0000 7f1dd17fa640 1 -- 172.21.15.6:0/3146888266 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0016020 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.723+0000 7f1dd17fa640 1 -- 172.21.15.6:0/3146888266 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dc0012870 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.723+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd406c520 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.723+0000 7f1dd17fa640 1 -- 172.21.15.6:0/3146888266 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0012c60 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.723+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd406cae0 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.723+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd409fa90 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.724+0000 7f1dd17fa640 1 -- 172.21.15.6:0/3146888266 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1dc0019020 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.724+0000 7f1dd17fa640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1db806bc00 0x7f1db806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.724+0000 7f1dd17fa640 1 -- 172.21.15.6:0/3146888266 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f1dc008acc0 con 0x7f1dd40d1d90 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.725+0000 7f1dd8b0b640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1db806bc00 0x7f1db806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.725+0000 7f1dd8b0b640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1db806bc00 0x7f1db806e0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800aa20 tx=0x7f1dc8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:21.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.729+0000 7f1dd17fa640 1 -- 172.21.15.6:0/3146888266 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1dc0020020 con 0x7f1dd40d1d90 2024-03-18T14:43:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.911+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1dd40ca410 con 0x7f1dd40d1d90 2024-03-18T14:43:21.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.912+0000 7f1dd17fa640 1 -- 172.21.15.6:0/3146888266 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+18707 (secure 0 0 0) 0x7f1dc0058e30 con 0x7f1dd40d1d90 2024-03-18T14:43:21.914 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:21.914 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:17.388447+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1db806bc00 msgr2=0x7f1db806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1db806bc00 0x7f1db806e0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800aa20 tx=0x7f1dc8008040 comp rx=0 tx=0).stop 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40d1d90 msgr2=0x7f1dd406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40d1d90 0x7f1dd406d510 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000c460 tx=0x7f1dc000c920 comp rx=0 tx=0).stop 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 shutdown_connections 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1db806bc00 0x7f1db806e0c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dd415bd40 0x7f1dd406da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dd40d1d90 0x7f1dd406d510 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 --2- 172.21.15.6:0/3146888266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1dd40cfea0 0x7f1dd4155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:21.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 >> 172.21.15.6:0/3146888266 conn(0x7f1dd409ae20 msgr2=0x7f1dd40713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:21.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 shutdown_connections 2024-03-18T14:43:21.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:21.914+0000 7f1ddad6f640 1 -- 172.21.15.6:0/3146888266 wait complete. 2024-03-18T14:43:21.927 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:42:54.939648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:42:55.638120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-18T14:43:03.985735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:21.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:22.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.041+0000 7faa54c9a640 1 Processor -- start 2024-03-18T14:43:22.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.041+0000 7faa54c9a640 1 -- start start 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa54c9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa50143ed0 0x7faa500cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa54c9a640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faa501443c0 0x7faa500b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa54c9a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5015f6c0 con 0x7faa500b0cd0 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa54c9a640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7faa500cf7d0 con 0x7faa50143ed0 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa54c9a640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7faa5015f230 con 0x7faa501443c0 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa4e59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa50143ed0 0x7faa500cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa4ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa500b0cd0 0x7faa50071ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56282/0 (socket says 172.21.15.6:56282) 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa4ed9d640 1 -- 172.21.15.6:0/2617117017 learned_addr learned my addr 172.21.15.6:0/2617117017 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa47fff640 1 --2- 172.21.15.6:0/2617117017 >> v2:172.21.15.92:3300/0 conn(0x7faa501443c0 0x7faa500b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa4e59c640 1 -- 172.21.15.6:0/2617117017 >> v2:172.21.15.92:3300/0 conn(0x7faa501443c0 msgr2=0x7faa500b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa4e59c640 1 --2- 172.21.15.6:0/2617117017 >> v2:172.21.15.92:3300/0 conn(0x7faa501443c0 0x7faa500b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa4e59c640 1 -- 172.21.15.6:0/2617117017 >> v1:172.21.15.6:6789/0 conn(0x7faa500b0cd0 legacy=0x7faa50071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.042+0000 7faa4e59c640 1 -- 172.21.15.6:0/2617117017 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5015e900 con 0x7faa50143ed0 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.043+0000 7faa4e59c640 1 --2- 172.21.15.6:0/2617117017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa50143ed0 0x7faa500cee80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7faa5015e6c0 tx=0x7faa4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea275c73d3cec4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.043+0000 7faa4dd9b640 1 -- 172.21.15.6:0/2617117017 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40035020 con 0x7faa50143ed0 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.043+0000 7faa4dd9b640 1 -- 172.21.15.6:0/2617117017 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa40031830 con 0x7faa50143ed0 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.043+0000 7faa4dd9b640 1 -- 172.21.15.6:0/2617117017 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40031c00 con 0x7faa50143ed0 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 -- 172.21.15.6:0/2617117017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa50143ed0 msgr2=0x7faa500cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 --2- 172.21.15.6:0/2617117017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa50143ed0 0x7faa500cee80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7faa5015e6c0 tx=0x7faa4002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 -- 172.21.15.6:0/2617117017 shutdown_connections 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 --2- 172.21.15.6:0/2617117017 >> v2:172.21.15.92:3300/0 conn(0x7faa501443c0 0x7faa500b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 --2- 172.21.15.6:0/2617117017 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa50143ed0 0x7faa500cee80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 -- 172.21.15.6:0/2617117017 >> 172.21.15.6:0/2617117017 conn(0x7faa5006b1c0 msgr2=0x7faa5006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 -- 172.21.15.6:0/2617117017 shutdown_connections 2024-03-18T14:43:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.044+0000 7faa54c9a640 1 -- 172.21.15.6:0/2617117017 wait complete. 2024-03-18T14:43:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.046+0000 7faa54c9a640 1 Processor -- start 2024-03-18T14:43:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.046+0000 7faa54c9a640 1 -- start start 2024-03-18T14:43:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa500b0cd0 0x7faa501e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa50143ed0 0x7faa500a87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa501443c0 0x7faa5015a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faa500b4880 con 0x7faa501443c0 2024-03-18T14:43:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faa5015eec0 con 0x7faa50143ed0 2024-03-18T14:43:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faa500cf3c0 con 0x7faa500b0cd0 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa501443c0 0x7faa5015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa501443c0 0x7faa5015a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38364/0 (socket says 172.21.15.6:38364) 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 -- 172.21.15.6:0/3404992704 learned_addr learned my addr 172.21.15.6:0/3404992704 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa47fff640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa50143ed0 0x7faa500a87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4e59c640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa500b0cd0 0x7faa501e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 -- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa500b0cd0 msgr2=0x7faa501e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa500b0cd0 0x7faa501e6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 -- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa50143ed0 msgr2=0x7faa500a87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa50143ed0 0x7faa500a87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 -- 172.21.15.6:0/3404992704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa501f74d0 con 0x7faa501443c0 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa47fff640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa50143ed0 0x7faa500a87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa4ed9d640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa501443c0 0x7faa5015a030 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faa4800cab0 tx=0x7faa4800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa46ffd640 1 -- 172.21.15.6:0/3404992704 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48017070 con 0x7faa501443c0 2024-03-18T14:43:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa46ffd640 1 -- 172.21.15.6:0/3404992704 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4800abd0 con 0x7faa501443c0 2024-03-18T14:43:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa46ffd640 1 -- 172.21.15.6:0/3404992704 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48012400 con 0x7faa501443c0 2024-03-18T14:43:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa501f77b0 con 0x7faa501443c0 2024-03-18T14:43:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.047+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa501f8050 con 0x7faa501443c0 2024-03-18T14:43:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.049+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa500751b0 con 0x7faa501443c0 2024-03-18T14:43:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.049+0000 7faa46ffd640 1 -- 172.21.15.6:0/3404992704 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faa4801a070 con 0x7faa501443c0 2024-03-18T14:43:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.049+0000 7faa46ffd640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa3006bbd0 0x7faa3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.049+0000 7faa46ffd640 1 -- 172.21.15.6:0/3404992704 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7faa4808a950 con 0x7faa501443c0 2024-03-18T14:43:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.050+0000 7faa4e59c640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa3006bbd0 0x7faa3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.050+0000 7faa4e59c640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa3006bbd0 0x7faa3006e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faa40036000 tx=0x7faa4003c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.053+0000 7faa46ffd640 1 -- 172.21.15.6:0/3404992704 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faa48058ac0 con 0x7faa501443c0 2024-03-18T14:43:22.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.236+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa500cc170 con 0x7faa501443c0 2024-03-18T14:43:22.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.236+0000 7faa46ffd640 1 -- 172.21.15.6:0/3404992704 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7faa4805cb60 con 0x7faa501443c0 2024-03-18T14:43:22.238 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa3006bbd0 msgr2=0x7faa3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa3006bbd0 0x7faa3006e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faa40036000 tx=0x7faa4003c040 comp rx=0 tx=0).stop 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa501443c0 msgr2=0x7faa5015a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa501443c0 0x7faa5015a030 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faa4800cab0 tx=0x7faa4800cf70 comp rx=0 tx=0).stop 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 shutdown_connections 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa3006bbd0 0x7faa3006e090 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa501443c0 0x7faa5015a030 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa50143ed0 0x7faa500a87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 --2- 172.21.15.6:0/3404992704 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa500b0cd0 0x7faa501e6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 >> 172.21.15.6:0/3404992704 conn(0x7faa5006b1c0 msgr2=0x7faa500b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 shutdown_connections 2024-03-18T14:43:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.238+0000 7faa54c9a640 1 -- 172.21.15.6:0/3404992704 wait complete. 2024-03-18T14:43:22.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.354+0000 7f7462afe640 1 Processor -- start 2024-03-18T14:43:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.354+0000 7f7462afe640 1 -- start start 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f7462afe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0ce9d0 0x7f745c0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f7462afe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f745c0d2510 0x7f745c0d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f7462afe640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f745c0d2970 con 0x7f745c0d0560 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f7462afe640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f745c0d09d0 con 0x7f745c0d2510 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f7462afe640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f745c0cd8d0 con 0x7f745c0ce9d0 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f746109b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f745c0d0560 0x7f745c15b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56288/0 (socket says 172.21.15.6:56288) 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f745bfff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f745c0d2510 0x7f745c0d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f746109b640 1 -- 172.21.15.6:0/2344179118 learned_addr learned my addr 172.21.15.6:0/2344179118 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f746089a640 1 --2- 172.21.15.6:0/2344179118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0ce9d0 0x7f745c0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f745b7fe640 1 -- 172.21.15.6:0/2344179118 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728519881 0 0) 0x7f745c0d2970 con 0x7f745c0d0560 2024-03-18T14:43:22.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f745b7fe640 1 -- 172.21.15.6:0/2344179118 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f744c003520 con 0x7f745c0d0560 2024-03-18T14:43:22.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f745bfff640 1 -- 172.21.15.6:0/2344179118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0ce9d0 msgr2=0x7f745c0cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:22.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f745bfff640 1 --2- 172.21.15.6:0/2344179118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0ce9d0 0x7f745c0cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f745bfff640 1 -- 172.21.15.6:0/2344179118 >> v1:172.21.15.6:6789/0 conn(0x7f745c0d0560 legacy=0x7f745c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.355+0000 7f745bfff640 1 -- 172.21.15.6:0/2344179118 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745c0cfac0 con 0x7f745c0d2510 2024-03-18T14:43:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.356+0000 7f745bfff640 1 --2- 172.21.15.6:0/2344179118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0d2510 0x7f745c0d0020 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f74480097a0 tx=0x7f744802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae2257c3e915ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.356+0000 7f745b7fe640 1 -- 172.21.15.6:0/2344179118 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7448031780 con 0x7f745c0d2510 2024-03-18T14:43:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.356+0000 7f745b7fe640 1 -- 172.21.15.6:0/2344179118 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7448037070 con 0x7f745c0d2510 2024-03-18T14:43:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.356+0000 7f745b7fe640 1 -- 172.21.15.6:0/2344179118 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7448031b20 con 0x7f745c0d2510 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 -- 172.21.15.6:0/2344179118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0d2510 msgr2=0x7f745c0d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 --2- 172.21.15.6:0/2344179118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0d2510 0x7f745c0d0020 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f74480097a0 tx=0x7f744802b820 comp rx=0 tx=0).stop 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 -- 172.21.15.6:0/2344179118 shutdown_connections 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 --2- 172.21.15.6:0/2344179118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0d2510 0x7f745c0d0020 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 --2- 172.21.15.6:0/2344179118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0ce9d0 0x7f745c0cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 -- 172.21.15.6:0/2344179118 >> 172.21.15.6:0/2344179118 conn(0x7f745c08b0c0 msgr2=0x7f745c0d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 -- 172.21.15.6:0/2344179118 shutdown_connections 2024-03-18T14:43:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.357+0000 7f7462afe640 1 -- 172.21.15.6:0/2344179118 wait complete. 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.358+0000 7f7462afe640 1 Processor -- start 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.359+0000 7f7462afe640 1 -- start start 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f7462afe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0ce9d0 0x7f745c158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f7462afe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0d0560 0x7f745c155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f7462afe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f745c0d2510 0x7f745c0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f7462afe640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f745c0cd280 con 0x7f745c0ce9d0 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f7462afe640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f745c0ccfa0 con 0x7f745c0d2510 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f7462afe640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f745c0cd560 con 0x7f745c0d0560 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0ce9d0 0x7f745c158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0ce9d0 0x7f745c158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38380/0 (socket says 172.21.15.6:38380) 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 -- 172.21.15.6:0/918264601 learned_addr learned my addr 172.21.15.6:0/918264601 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f745bfff640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0d0560 0x7f745c155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746109b640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f745c0d2510 0x7f745c0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 -- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0d0560 msgr2=0x7f745c155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0d0560 0x7f745c155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 -- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f745c0d2510 msgr2=0x7f745c0c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f745c0d2510 0x7f745c0c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 -- 172.21.15.6:0/918264601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745c153b70 con 0x7f745c0ce9d0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f745bfff640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0d0560 0x7f745c155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.360+0000 7f746089a640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0ce9d0 0x7f745c158810 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f745000e930 tx=0x7f745000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.361+0000 7f74597fa640 1 -- 172.21.15.6:0/918264601 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450017070 con 0x7f745c0ce9d0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.361+0000 7f74597fa640 1 -- 172.21.15.6:0/918264601 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745000ca00 con 0x7f745c0ce9d0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.361+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f745c0b50d0 con 0x7f745c0ce9d0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.361+0000 7f74597fa640 1 -- 172.21.15.6:0/918264601 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745000cdb0 con 0x7f745c0ce9d0 2024-03-18T14:43:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.361+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f745c158d50 con 0x7f745c0ce9d0 2024-03-18T14:43:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.362+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f745c0751b0 con 0x7f745c0ce9d0 2024-03-18T14:43:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.363+0000 7f74597fa640 1 -- 172.21.15.6:0/918264601 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f745001a070 con 0x7f745c0ce9d0 2024-03-18T14:43:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.363+0000 7f74597fa640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f744006bbd0 0x7f744006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.363+0000 7f74597fa640 1 -- 172.21.15.6:0/918264601 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f745008b130 con 0x7f745c0ce9d0 2024-03-18T14:43:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.363+0000 7f745bfff640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f744006bbd0 0x7f744006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.364+0000 7f745bfff640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f744006bbd0 0x7f744006e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7448036000 tx=0x7f744803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.368+0000 7f74597fa640 1 -- 172.21.15.6:0/918264601 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7450059a40 con 0x7f745c0ce9d0 2024-03-18T14:43:22.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.549+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f745c0cfe00 con 0x7f745c0ce9d0 2024-03-18T14:43:22.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.549+0000 7f74597fa640 1 -- 172.21.15.6:0/918264601 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7f7450059340 con 0x7f745c0ce9d0 2024-03-18T14:43:22.551 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f744006bbd0 msgr2=0x7f744006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f744006bbd0 0x7f744006e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7448036000 tx=0x7f744803d040 comp rx=0 tx=0).stop 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0ce9d0 msgr2=0x7f745c158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0ce9d0 0x7f745c158810 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f745000e930 tx=0x7f745000edf0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 shutdown_connections 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f744006bbd0 0x7f744006e090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f745c0d2510 0x7f745c0c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f745c0d0560 0x7f745c155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 --2- 172.21.15.6:0/918264601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c0ce9d0 0x7f745c158810 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 >> 172.21.15.6:0/918264601 conn(0x7f745c08b0c0 msgr2=0x7f745c0ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:22.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 shutdown_connections 2024-03-18T14:43:22.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.551+0000 7f7462afe640 1 -- 172.21.15.6:0/918264601 wait complete. 2024-03-18T14:43:22.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:22.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.667+0000 7f796eeac640 1 Processor -- start 2024-03-18T14:43:22.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.667+0000 7f796eeac640 1 -- start start 2024-03-18T14:43:22.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.668+0000 7f796eeac640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f796815e030 0x7f79680d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.668+0000 7f796eeac640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79680b65a0 con 0x7f79680b6140 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.668+0000 7f796eeac640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79680d2fd0 con 0x7f79680d0450 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.668+0000 7f796eeac640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f796815e410 con 0x7f796815e030 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.668+0000 7f7967fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f79680b6140 0x7f79680ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56292/0 (socket says 172.21.15.6:56292) 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.668+0000 7f7967fff640 1 -- 172.21.15.6:0/397277352 learned_addr learned my addr 172.21.15.6:0/397277352 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.668+0000 7f796cc48640 1 --2- 172.21.15.6:0/397277352 >> v2:172.21.15.92:3300/0 conn(0x7f796815e030 0x7f79680d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74214335 0 0) 0x7f79680b65a0 con 0x7f79680b6140 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7958003520 con 0x7f79680b6140 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500026476 0 0) 0x7f79680d2fd0 con 0x7f79680d0450 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79680b65a0 con 0x7f79680d0450 2024-03-18T14:43:22.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3713613229 0 0) 0x7f7958003520 con 0x7f79680b6140 2024-03-18T14:43:22.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f796cc48640 1 -- 172.21.15.6:0/397277352 >> v1:172.21.15.92:6789/0 conn(0x7f79680d0450 legacy=0x7f79680d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f796cc48640 1 -- 172.21.15.6:0/397277352 >> v1:172.21.15.6:6789/0 conn(0x7f79680b6140 legacy=0x7f79680ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.669+0000 7f796cc48640 1 -- 172.21.15.6:0/397277352 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79680d3290 con 0x7f796815e030 2024-03-18T14:43:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f796cc48640 1 --2- 172.21.15.6:0/397277352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796815e030 0x7f79680d3570 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f795c0097a0 tx=0x7f795c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9ffd58aee3d4b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c038070 con 0x7f796815e030 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795c031500 con 0x7f796815e030 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f79677fe640 1 -- 172.21.15.6:0/397277352 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954001670 con 0x7f796815e030 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f796eeac640 1 -- 172.21.15.6:0/397277352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796815e030 msgr2=0x7f79680d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f796eeac640 1 --2- 172.21.15.6:0/397277352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796815e030 0x7f79680d3570 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f795c0097a0 tx=0x7f795c02b820 comp rx=0 tx=0).stop 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f796eeac640 1 -- 172.21.15.6:0/397277352 shutdown_connections 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f796eeac640 1 --2- 172.21.15.6:0/397277352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796815e030 0x7f79680d3570 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.670+0000 7f796eeac640 1 -- 172.21.15.6:0/397277352 >> 172.21.15.6:0/397277352 conn(0x7f796806c8c0 msgr2=0x7f796806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.671+0000 7f796eeac640 1 -- 172.21.15.6:0/397277352 shutdown_connections 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.671+0000 7f796eeac640 1 -- 172.21.15.6:0/397277352 wait complete. 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.672+0000 7f796eeac640 1 Processor -- start 2024-03-18T14:43:22.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.672+0000 7f796eeac640 1 -- start start 2024-03-18T14:43:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796eeac640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79680b6140 0x7f796814f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796eeac640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79680d0450 0x7f7968160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796eeac640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796815e030 0x7f796815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796eeac640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f79680ce830 con 0x7f796815e030 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796eeac640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f79680ce300 con 0x7f79680b6140 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796eeac640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7968161a90 con 0x7f79680d0450 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796815e030 0x7f796815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796815e030 0x7f796815b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38392/0 (socket says 172.21.15.6:38392) 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 -- 172.21.15.6:0/2249572839 learned_addr learned my addr 172.21.15.6:0/2249572839 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f7967fff640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79680d0450 0x7f7968160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796cc48640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79680b6140 0x7f796814f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 -- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79680d0450 msgr2=0x7f7968160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79680d0450 0x7f7968160f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 -- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79680b6140 msgr2=0x7f796814f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79680b6140 0x7f796814f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 -- 172.21.15.6:0/2249572839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796814f5b0 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796d449640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796815e030 0x7f796815b9e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f796000c790 tx=0x7f796000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f79657fa640 1 -- 172.21.15.6:0/2249572839 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960003b70 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f79657fa640 1 -- 172.21.15.6:0/2249572839 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7960003d10 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.673+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79680c8870 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.674+0000 7f79657fa640 1 -- 172.21.15.6:0/2249572839 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796000aa50 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.674+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79680c8e30 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.675+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79680751b0 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.675+0000 7f79657fa640 1 -- 172.21.15.6:0/2249572839 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f796000abf0 con 0x7f796815e030 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.675+0000 7f79657fa640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f794c06bc00 0x7f794c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:22.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.675+0000 7f79657fa640 1 -- 172.21.15.6:0/2249572839 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f796008ac00 con 0x7f796815e030 2024-03-18T14:43:22.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.675+0000 7f796cc48640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f794c06bc00 0x7f794c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:22.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.676+0000 7f796cc48640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f794c06bc00 0x7f794c06e0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f795c035000 tx=0x7f795c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:22.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.680+0000 7f79657fa640 1 -- 172.21.15.6:0/2249572839 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7960091080 con 0x7f796815e030 2024-03-18T14:43:22.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.890+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f79680b86c0 con 0x7f796815e030 2024-03-18T14:43:22.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.891+0000 7f79657fa640 1 -- 172.21.15.6:0/2249572839 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7f7960058d70 con 0x7f796815e030 2024-03-18T14:43:22.892 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f794c06bc00 msgr2=0x7f794c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f794c06bc00 0x7f794c06e0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f795c035000 tx=0x7f795c008030 comp rx=0 tx=0).stop 2024-03-18T14:43:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796815e030 msgr2=0x7f796815b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796815e030 0x7f796815b9e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f796000c790 tx=0x7f796000cc50 comp rx=0 tx=0).stop 2024-03-18T14:43:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 shutdown_connections 2024-03-18T14:43:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f794c06bc00 0x7f794c06e0c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f796815e030 0x7f796815b9e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79680d0450 0x7f7968160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 --2- 172.21.15.6:0/2249572839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79680b6140 0x7f796814f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.893+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 >> 172.21.15.6:0/2249572839 conn(0x7f796806c8c0 msgr2=0x7f79680b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.894+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 shutdown_connections 2024-03-18T14:43:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:22.894+0000 7f796eeac640 1 -- 172.21.15.6:0/2249572839 wait complete. 2024-03-18T14:43:22.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-18T14:43:23.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.009+0000 7f209acc6640 1 Processor -- start 2024-03-18T14:43:23.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.009+0000 7f209acc6640 1 -- start start 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f209acc6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f209415e030 0x7f20940d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f209acc6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20940b6140 0x7f20940ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f209acc6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940d0450 0x7f20940d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f209acc6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f209415e410 con 0x7f209415e030 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f209acc6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f20940ce300 con 0x7f20940b6140 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f209acc6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f20940d0e20 con 0x7f20940d0450 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f209415e030 0x7f20940d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f209415e030 0x7f20940d3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38408/0 (socket says 172.21.15.6:38408) 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 -- 172.21.15.6:0/854148267 learned_addr learned my addr 172.21.15.6:0/854148267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2093fff640 1 --2- 172.21.15.6:0/854148267 >> v2:172.21.15.92:3300/0 conn(0x7f20940b6140 0x7f20940ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2099263640 1 --2- 172.21.15.6:0/854148267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940d0450 0x7f20940d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 -- 172.21.15.6:0/854148267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940d0450 msgr2=0x7f20940d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 --2- 172.21.15.6:0/854148267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940d0450 0x7f20940d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 -- 172.21.15.6:0/854148267 >> v2:172.21.15.92:3300/0 conn(0x7f20940b6140 msgr2=0x7f20940ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 --2- 172.21.15.6:0/854148267 >> v2:172.21.15.92:3300/0 conn(0x7f20940b6140 0x7f20940ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2098a62640 1 -- 172.21.15.6:0/854148267 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20940b65a0 con 0x7f209415e030 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.010+0000 7f2093fff640 1 --2- 172.21.15.6:0/854148267 >> v2:172.21.15.92:3300/0 conn(0x7f20940b6140 0x7f20940ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.011+0000 7f2098a62640 1 --2- 172.21.15.6:0/854148267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20940d3570 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f20940b6840 tx=0x7f208802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c89006b3ad33ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.011+0000 7f20937fe640 1 -- 172.21.15.6:0/854148267 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088035020 con 0x7f209415e030 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.011+0000 7f20937fe640 1 -- 172.21.15.6:0/854148267 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2088031870 con 0x7f209415e030 2024-03-18T14:43:23.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.011+0000 7f20937fe640 1 -- 172.21.15.6:0/854148267 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088031c40 con 0x7f209415e030 2024-03-18T14:43:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 -- 172.21.15.6:0/854148267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 msgr2=0x7f20940d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 --2- 172.21.15.6:0/854148267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20940d3570 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f20940b6840 tx=0x7f208802b910 comp rx=0 tx=0).stop 2024-03-18T14:43:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 -- 172.21.15.6:0/854148267 shutdown_connections 2024-03-18T14:43:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 --2- 172.21.15.6:0/854148267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940d0450 0x7f20940d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 --2- 172.21.15.6:0/854148267 >> v2:172.21.15.92:3300/0 conn(0x7f20940b6140 0x7f20940ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 --2- 172.21.15.6:0/854148267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20940d3570 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 -- 172.21.15.6:0/854148267 >> 172.21.15.6:0/854148267 conn(0x7f209406c8c0 msgr2=0x7f209406ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 -- 172.21.15.6:0/854148267 shutdown_connections 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.012+0000 7f209acc6640 1 -- 172.21.15.6:0/854148267 wait complete. 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.013+0000 7f209acc6640 1 Processor -- start 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.013+0000 7f209acc6640 1 -- start start 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f209acc6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940b6140 0x7f20940c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f209acc6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20940d0450 0x7f20941f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f209acc6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20941f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f209acc6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f20940ce550 con 0x7f209415e030 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f209acc6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f20940d3290 con 0x7f20940d0450 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f209acc6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f20940ce830 con 0x7f20940b6140 2024-03-18T14:43:23.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f2099263640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20941f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f2099263640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20941f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38414/0 (socket says 172.21.15.6:38414) 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f2099263640 1 -- 172.21.15.6:0/1040342948 learned_addr learned my addr 172.21.15.6:0/1040342948 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.014+0000 7f2098a62640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940b6140 0x7f20940c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f2093fff640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20940d0450 0x7f20941f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f2099263640 1 -- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940b6140 msgr2=0x7f20940c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f2099263640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940b6140 0x7f20940c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f2099263640 1 -- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20940d0450 msgr2=0x7f20941f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f2099263640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20940d0450 0x7f20941f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f2099263640 1 -- 172.21.15.6:0/1040342948 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20940b4240 con 0x7f209415e030 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f2099263640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20941f7700 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f208c007e70 tx=0x7f208c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f20917fa640 1 -- 172.21.15.6:0/1040342948 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c014620 con 0x7f209415e030 2024-03-18T14:43:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f20917fa640 1 -- 172.21.15.6:0/1040342948 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f208c019070 con 0x7f209415e030 2024-03-18T14:43:23.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20940b4520 con 0x7f209415e030 2024-03-18T14:43:23.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f20917fa640 1 -- 172.21.15.6:0/1040342948 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c014a20 con 0x7f209415e030 2024-03-18T14:43:23.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.015+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20940b4ae0 con 0x7f209415e030 2024-03-18T14:43:23.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.016+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20940751b0 con 0x7f209415e030 2024-03-18T14:43:23.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.017+0000 7f20917fa640 1 -- 172.21.15.6:0/1040342948 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f208c014bc0 con 0x7f209415e030 2024-03-18T14:43:23.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.017+0000 7f20917fa640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f207806bbd0 0x7f207806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.017+0000 7f20917fa640 1 -- 172.21.15.6:0/1040342948 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f208c08b4f0 con 0x7f209415e030 2024-03-18T14:43:23.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.017+0000 7f2098a62640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f207806bbd0 0x7f207806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.018+0000 7f2098a62640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f207806bbd0 0x7f207806e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2088036000 tx=0x7f208803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.021+0000 7f20917fa640 1 -- 172.21.15.6:0/1040342948 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f208c0596f0 con 0x7f209415e030 2024-03-18T14:43:23.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.203+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f20940d19f0 con 0x7f209415e030 2024-03-18T14:43:23.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.204+0000 7f20917fa640 1 -- 172.21.15.6:0/1040342948 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v44) ==== 97+0+11 (secure 0 0 0) 0x7f208c05d790 con 0x7f209415e030 2024-03-18T14:43:23.205 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:23.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f207806bbd0 msgr2=0x7f207806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f207806bbd0 0x7f207806e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2088036000 tx=0x7f208803c040 comp rx=0 tx=0).stop 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 msgr2=0x7f20941f7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20941f7700 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f208c007e70 tx=0x7f208c00e4b0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 shutdown_connections 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f207806bbd0 0x7f207806e090 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f209415e030 0x7f20941f7700 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20940d0450 0x7f20941f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 --2- 172.21.15.6:0/1040342948 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20940b6140 0x7f20940c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 >> 172.21.15.6:0/1040342948 conn(0x7f209406c8c0 msgr2=0x7f20940d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 shutdown_connections 2024-03-18T14:43:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.206+0000 7f209acc6640 1 -- 172.21.15.6:0/1040342948 wait complete. 2024-03-18T14:43:23.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.318+0000 7f3addc19640 1 Processor -- start 2024-03-18T14:43:23.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.318+0000 7f3addc19640 1 -- start start 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3addc19640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3ad806c4e0 0x7f3ad80d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3addc19640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ad80b20d0 0x7f3ad80d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3addc19640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ad806ca50 con 0x7f3ad815e010 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3addc19640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3ad806ce10 con 0x7f3ad80b20d0 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3addc19640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f3ad806d0f0 con 0x7f3ad806c4e0 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad7fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ad80b20d0 0x7f3ad80d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad7fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ad80b20d0 0x7f3ad80d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38418/0 (socket says 172.21.15.6:38418) 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad7fff640 1 -- 172.21.15.6:0/1034280498 learned_addr learned my addr 172.21.15.6:0/1034280498 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad6ffd640 1 --2- 172.21.15.6:0/1034280498 >> v2:172.21.15.92:3300/0 conn(0x7f3ad806c4e0 0x7f3ad80d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad7fff640 1 -- 172.21.15.6:0/1034280498 >> v2:172.21.15.92:3300/0 conn(0x7f3ad806c4e0 msgr2=0x7f3ad80d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:23.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad7fff640 1 --2- 172.21.15.6:0/1034280498 >> v2:172.21.15.92:3300/0 conn(0x7f3ad806c4e0 0x7f3ad80d0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad7fff640 1 -- 172.21.15.6:0/1034280498 >> v1:172.21.15.92:6789/0 conn(0x7f3ad815e010 legacy=0x7f3ad80b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.319+0000 7f3ad7fff640 1 -- 172.21.15.6:0/1034280498 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad80b34f0 con 0x7f3ad80b20d0 2024-03-18T14:43:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.320+0000 7f3ad7fff640 1 --2- 172.21.15.6:0/1034280498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad80b20d0 0x7f3ad80d1900 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ad806e480 tx=0x7f3ad002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450ba17568fcd83d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.320+0000 7f3ad67fc640 1 -- 172.21.15.6:0/1034280498 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad002ca80 con 0x7f3ad80b20d0 2024-03-18T14:43:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.320+0000 7f3ad67fc640 1 -- 172.21.15.6:0/1034280498 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ad002cc20 con 0x7f3ad80b20d0 2024-03-18T14:43:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.320+0000 7f3ad67fc640 1 -- 172.21.15.6:0/1034280498 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad00343f0 con 0x7f3ad80b20d0 2024-03-18T14:43:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 -- 172.21.15.6:0/1034280498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad80b20d0 msgr2=0x7f3ad80d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 --2- 172.21.15.6:0/1034280498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad80b20d0 0x7f3ad80d1900 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ad806e480 tx=0x7f3ad002bee0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 -- 172.21.15.6:0/1034280498 shutdown_connections 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 --2- 172.21.15.6:0/1034280498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad80b20d0 0x7f3ad80d1900 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 --2- 172.21.15.6:0/1034280498 >> v2:172.21.15.92:3300/0 conn(0x7f3ad806c4e0 0x7f3ad80d0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 -- 172.21.15.6:0/1034280498 >> 172.21.15.6:0/1034280498 conn(0x7f3ad808b320 msgr2=0x7f3ad808b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 -- 172.21.15.6:0/1034280498 shutdown_connections 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.321+0000 7f3addc19640 1 -- 172.21.15.6:0/1034280498 wait complete. 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.322+0000 7f3addc19640 1 Processor -- start 2024-03-18T14:43:23.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.322+0000 7f3addc19640 1 -- start start 2024-03-18T14:43:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3addc19640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad806c4e0 0x7f3ad80b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3addc19640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ad80b20d0 0x7f3ad80b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3addc19640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ad815e010 0x7f3ad8147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3addc19640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad815e570 con 0x7f3ad806c4e0 2024-03-18T14:43:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3addc19640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad815e3f0 con 0x7f3ad80b20d0 2024-03-18T14:43:23.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3addc19640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f3ad80b2560 con 0x7f3ad815e010 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad806c4e0 0x7f3ad80b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad806c4e0 0x7f3ad80b5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38422/0 (socket says 172.21.15.6:38422) 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 -- 172.21.15.6:0/805899503 learned_addr learned my addr 172.21.15.6:0/805899503 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad6ffd640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ad80b20d0 0x7f3ad80b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad7fff640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ad815e010 0x7f3ad8147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 -- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ad815e010 msgr2=0x7f3ad8147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ad815e010 0x7f3ad8147d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 -- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ad80b20d0 msgr2=0x7f3ad80b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ad80b20d0 0x7f3ad80b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.323+0000 7f3ad77fe640 1 -- 172.21.15.6:0/805899503 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad80b65f0 con 0x7f3ad806c4e0 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.324+0000 7f3ad77fe640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad806c4e0 0x7f3ad80b5f10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c3c0 tx=0x7f3ac000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.324+0000 7f3acffff640 1 -- 172.21.15.6:0/805899503 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00052d0 con 0x7f3ad806c4e0 2024-03-18T14:43:23.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.324+0000 7f3acffff640 1 -- 172.21.15.6:0/805899503 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac0005490 con 0x7f3ad806c4e0 2024-03-18T14:43:23.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.324+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad80b68d0 con 0x7f3ad806c4e0 2024-03-18T14:43:23.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.324+0000 7f3acffff640 1 -- 172.21.15.6:0/805899503 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0007260 con 0x7f3ad806c4e0 2024-03-18T14:43:23.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.324+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad80b6e90 con 0x7f3ad806c4e0 2024-03-18T14:43:23.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.325+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad804f5b0 con 0x7f3ad806c4e0 2024-03-18T14:43:23.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.326+0000 7f3acffff640 1 -- 172.21.15.6:0/805899503 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3ac0007400 con 0x7f3ad806c4e0 2024-03-18T14:43:23.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.326+0000 7f3acffff640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ab806bbd0 0x7f3ab806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.326+0000 7f3acffff640 1 -- 172.21.15.6:0/805899503 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f3ac008a140 con 0x7f3ad806c4e0 2024-03-18T14:43:23.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.326+0000 7f3ad6ffd640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ab806bbd0 0x7f3ab806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.327+0000 7f3ad6ffd640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ab806bbd0 0x7f3ab806e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8071320 tx=0x7f3ac8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.330+0000 7f3acffff640 1 -- 172.21.15.6:0/805899503 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3ac0058340 con 0x7f3ad806c4e0 2024-03-18T14:43:23.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.528+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3ad80cee80 con 0x7f3ad806c4e0 2024-03-18T14:43:23.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.528+0000 7f3acffff640 1 -- 172.21.15.6:0/805899503 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) ==== 80+0+0 (secure 0 0 0) 0x7f3ac005c3e0 con 0x7f3ad806c4e0 2024-03-18T14:43:23.530 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:23.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ab806bbd0 msgr2=0x7f3ab806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ab806bbd0 0x7f3ab806e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8071320 tx=0x7f3ac8009190 comp rx=0 tx=0).stop 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad806c4e0 msgr2=0x7f3ad80b5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad806c4e0 0x7f3ad80b5f10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c3c0 tx=0x7f3ac000c880 comp rx=0 tx=0).stop 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 shutdown_connections 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f3ab806bbd0 0x7f3ab806e090 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ad815e010 0x7f3ad8147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ad80b20d0 0x7f3ad80b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 --2- 172.21.15.6:0/805899503 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ad806c4e0 0x7f3ad80b5f10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 >> 172.21.15.6:0/805899503 conn(0x7f3ad808b320 msgr2=0x7f3ad815d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 shutdown_connections 2024-03-18T14:43:23.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.530+0000 7f3addc19640 1 -- 172.21.15.6:0/805899503 wait complete. 2024-03-18T14:43:23.542 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:43:23.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:23.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:23.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[56794]: fuse finished with error 0 and tester_r 0 2024-03-18T14:43:23.569 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:23.587 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:43:23.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:23.587 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:43:23.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-18T14:43:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.792+0000 7f40d0481640 1 Processor -- start 2024-03-18T14:43:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.792+0000 7f40d0481640 1 -- start start 2024-03-18T14:43:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40d0481640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f40c80afac0 0x7f40c806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40d0481640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 0x7f40c80cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40d0481640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c80ce040 con 0x7f40c815e030 2024-03-18T14:43:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40d0481640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f40c815e410 con 0x7f40c80afac0 2024-03-18T14:43:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40d0481640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f40c80d11a0 con 0x7f40c806b700 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40ce21d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f40c815e030 0x7f40c80cd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59782/0 (socket says 172.21.15.6:59782) 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40ce21d640 1 -- 172.21.15.6:0/510823619 learned_addr learned my addr 172.21.15.6:0/510823619 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cda1c640 1 --2- 172.21.15.6:0/510823619 >> v2:172.21.15.92:3300/0 conn(0x7f40c80afac0 0x7f40c806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cea1e640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 0x7f40c80cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 65936845 0 0) 0x7f40c4001110 con 0x7f40c815e030 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b4003520 con 0x7f40c815e030 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3646440640 0 0) 0x7f40b4003520 con 0x7f40c815e030 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 msgr2=0x7f40c80cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 0x7f40c80cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 >> v2:172.21.15.92:3300/0 conn(0x7f40c80afac0 msgr2=0x7f40c806b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 --2- 172.21.15.6:0/510823619 >> v2:172.21.15.92:3300/0 conn(0x7f40c80afac0 0x7f40c806b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c80d0870 con 0x7f40c815e030 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f40c4001840 con 0x7f40c815e030 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 >> v1:172.21.15.6:6789/0 conn(0x7f40c815e030 legacy=0x7f40c80cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 0x7f40b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f40c815e410 con 0x7f40b40039b0 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40c80d11a0 con 0x7f40b4004800 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.793+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f40b400b9f0 con 0x7f40c806b700 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40ce21d640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40cda1c640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40cea1e640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 0x7f40b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40ce21d640 1 -- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 msgr2=0x7f40b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40ce21d640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c806b700 0x7f40b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40ce21d640 1 -- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40b4004800 msgr2=0x7f40b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40ce21d640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.794+0000 7f40ce21d640 1 -- 172.21.15.6:0/510823619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4003520 con 0x7f40b40039b0 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40ce21d640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c3e0 tx=0x7f40b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be093d08a8ea0fb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c400c0e0 con 0x7f40b40039b0 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c4030070 con 0x7f40b40039b0 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40cd21b640 1 -- 172.21.15.6:0/510823619 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c40058d0 con 0x7f40b40039b0 2024-03-18T14:43:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40d0481640 1 -- 172.21.15.6:0/510823619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 msgr2=0x7f40b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40d0481640 1 --2- 172.21.15.6:0/510823619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c3e0 tx=0x7f40b4005830 comp rx=0 tx=0).stop 2024-03-18T14:43:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40cea1e640 1 -- 172.21.15.6:0/510823619 reap_dead start 2024-03-18T14:43:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40d0481640 1 -- 172.21.15.6:0/510823619 shutdown_connections 2024-03-18T14:43:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40d0481640 1 -- 172.21.15.6:0/510823619 >> 172.21.15.6:0/510823619 conn(0x7f40c8099a50 msgr2=0x7f40c8099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40d0481640 1 -- 172.21.15.6:0/510823619 shutdown_connections 2024-03-18T14:43:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.795+0000 7f40d0481640 1 -- 172.21.15.6:0/510823619 wait complete. 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 Processor -- start 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 -- start start 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40c81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c80b0320 0x7f40c815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c815c4e0 0x7f40c81f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f40c80d1630 con 0x7f40b40039b0 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40c80d0e30 con 0x7f40c815c4e0 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40d0481640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f40c81f7da0 con 0x7f40c80b0320 2024-03-18T14:43:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40ce21d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40c81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40ce21d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40c81559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34720/0 (socket says 172.21.15.6:34720) 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.797+0000 7f40ce21d640 1 -- 172.21.15.6:0/4234502978 learned_addr learned my addr 172.21.15.6:0/4234502978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40cda1c640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c80b0320 0x7f40c815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40cea1e640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c815c4e0 0x7f40c81f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40ce21d640 1 -- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c80b0320 msgr2=0x7f40c815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40ce21d640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c80b0320 0x7f40c815bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40ce21d640 1 -- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c815c4e0 msgr2=0x7f40c81f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40ce21d640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c815c4e0 0x7f40c81f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40ce21d640 1 -- 172.21.15.6:0/4234502978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c815fa90 con 0x7f40b40039b0 2024-03-18T14:43:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40ce21d640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40c81559b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f40c402c7b0 tx=0x7f40c4001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40beffd640 1 -- 172.21.15.6:0/4234502978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4004bf0 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40beffd640 1 -- 172.21.15.6:0/4234502978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c4004d90 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c815fd70 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40beffd640 1 -- 172.21.15.6:0/4234502978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c40072b0 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.798+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c8160330 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.799+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c80751b0 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.800+0000 7f40beffd640 1 -- 172.21.15.6:0/4234502978 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f40c4007450 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.800+0000 7f40beffd640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f40ac06bbd0 0x7f40ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.800+0000 7f40beffd640 1 -- 172.21.15.6:0/4234502978 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f40c40ac970 con 0x7f40b40039b0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.800+0000 7f40cda1c640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f40ac06bbd0 0x7f40ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.801+0000 7f40cda1c640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f40ac06bbd0 0x7f40ac06e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f40c81f6f40 tx=0x7f40b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:23.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:23.804+0000 7f40beffd640 1 -- 172.21.15.6:0/4234502978 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f40c80ce040 con 0x7f40b40039b0 2024-03-18T14:43:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.057+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f40c815c8c0 con 0x7f40b40039b0 2024-03-18T14:43:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.059+0000 7f40beffd640 1 -- 172.21.15.6:0/4234502978 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v14) ==== 184+0+141 (secure 0 0 0) 0x7f40c40343d0 con 0x7f40b40039b0 2024-03-18T14:43:24.060 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:43:24.060 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:43:24.060 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-18T14:43:24.060 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:43:24.060 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-18T14:43:24.061 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:43:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f40ac06bbd0 msgr2=0x7f40ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f40ac06bbd0 0x7f40ac06e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f40c81f6f40 tx=0x7f40b8008040 comp rx=0 tx=0).stop 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 msgr2=0x7f40c81559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40c81559b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f40c402c7b0 tx=0x7f40c4001c10 comp rx=0 tx=0).stop 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 shutdown_connections 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f40ac06bbd0 0x7f40ac06e090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40c815c4e0 0x7f40c81f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40c80b0320 0x7f40c815bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 --2- 172.21.15.6:0/4234502978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40b40039b0 0x7f40c81559b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 >> 172.21.15.6:0/4234502978 conn(0x7f40c8099a50 msgr2=0x7f40c80cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 shutdown_connections 2024-03-18T14:43:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.061+0000 7f40d0481640 1 -- 172.21.15.6:0/4234502978 wait complete. 2024-03-18T14:43:24.073 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T14:43:24.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:43:24.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:43:24.157 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T14:43:24.157 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:43:24.157 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:24.157 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:24.172 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2314 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-18T14:43:24.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:43:24.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:24.172 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:43:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:43:24.226 INFO:teuthology.orchestra.run.smithi006.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-18T14:43:24.226 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:43:24.227 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:43:24.227 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:24.244 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:43:24.244 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:43:24.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:24.245 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T14:43:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:43:24.301 INFO:teuthology.orchestra.run.smithi092.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T14:43:24.302 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:43:24.302 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:24.302 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:24.316 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2314 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-18T14:43:24.317 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:43:24.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:24.317 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T14:43:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:43:24.371 INFO:teuthology.orchestra.run.smithi006.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T14:43:24.371 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:43:24.371 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:43:24.372 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:24.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:43:24.388 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:43:24.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:24.389 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T14:43:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:43:24.444 INFO:teuthology.orchestra.run.smithi092.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T14:43:24.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.572+0000 7f4404390640 1 Processor -- start 2024-03-18T14:43:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.572+0000 7f4404390640 1 -- start start 2024-03-18T14:43:24.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f4404390640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43fc0b1740 0x7f43fc0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f4404390640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43fc06b960 0x7f43fc0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f4404390640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43fc0ccd40 con 0x7f43fc0b1b20 2024-03-18T14:43:24.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f4404390640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f43fc06bdd0 con 0x7f43fc0b1740 2024-03-18T14:43:24.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f4404390640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f43fc0cd020 con 0x7f43fc06b960 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f440212c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43fc0b1740 0x7f43fc0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f440212c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43fc0b1740 0x7f43fc0cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34722/0 (socket says 172.21.15.6:34722) 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.573+0000 7f440212c640 1 -- 172.21.15.6:0/2149192213 learned_addr learned my addr 172.21.15.6:0/2149192213 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.574+0000 7f440292d640 1 --2- 172.21.15.6:0/2149192213 >> v2:172.21.15.92:3300/0 conn(0x7f43fc06b960 0x7f43fc0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.574+0000 7f440212c640 1 -- 172.21.15.6:0/2149192213 >> v2:172.21.15.92:3300/0 conn(0x7f43fc06b960 msgr2=0x7f43fc0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.574+0000 7f440212c640 1 --2- 172.21.15.6:0/2149192213 >> v2:172.21.15.92:3300/0 conn(0x7f43fc06b960 0x7f43fc0caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.574+0000 7f440212c640 1 -- 172.21.15.6:0/2149192213 >> v1:172.21.15.92:6789/0 conn(0x7f43fc0b1b20 legacy=0x7f43fc06b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.574+0000 7f440212c640 1 -- 172.21.15.6:0/2149192213 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc0cd300 con 0x7f43fc0b1740 2024-03-18T14:43:24.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.574+0000 7f440212c640 1 --2- 172.21.15.6:0/2149192213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1740 0x7f43fc0cdde0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0c7a20 tx=0x7f43f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5638373f3086d5e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.574+0000 7f440112a640 1 -- 172.21.15.6:0/2149192213 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8034070 con 0x7f43fc0b1740 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f440112a640 1 -- 172.21.15.6:0/2149192213 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f8030860 con 0x7f43fc0b1740 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f440112a640 1 -- 172.21.15.6:0/2149192213 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8030c30 con 0x7f43fc0b1740 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 -- 172.21.15.6:0/2149192213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1740 msgr2=0x7f43fc0cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 --2- 172.21.15.6:0/2149192213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1740 0x7f43fc0cdde0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0c7a20 tx=0x7f43f802b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 -- 172.21.15.6:0/2149192213 shutdown_connections 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 --2- 172.21.15.6:0/2149192213 >> v2:172.21.15.92:3300/0 conn(0x7f43fc06b960 0x7f43fc0caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 --2- 172.21.15.6:0/2149192213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1740 0x7f43fc0cdde0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 -- 172.21.15.6:0/2149192213 >> 172.21.15.6:0/2149192213 conn(0x7f43fc0d09a0 msgr2=0x7f43fc0d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 -- 172.21.15.6:0/2149192213 shutdown_connections 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.575+0000 7f4404390640 1 -- 172.21.15.6:0/2149192213 wait complete. 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.576+0000 7f4404390640 1 Processor -- start 2024-03-18T14:43:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.576+0000 7f4404390640 1 -- start start 2024-03-18T14:43:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f4404390640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43fc06b960 0x7f43fc0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f4404390640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43fc0b1740 0x7f43fc1dee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f4404390640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1b20 0x7f43fc0b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f4404390640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc0b1f90 con 0x7f43fc0b1b20 2024-03-18T14:43:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f4404390640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc0cd8c0 con 0x7f43fc06b960 2024-03-18T14:43:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f4404390640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f43fc0cb530 con 0x7f43fc0b1740 2024-03-18T14:43:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440292d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1b20 0x7f43fc0b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440292d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1b20 0x7f43fc0b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34736/0 (socket says 172.21.15.6:34736) 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440292d640 1 -- 172.21.15.6:0/2638917355 learned_addr learned my addr 172.21.15.6:0/2638917355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440212c640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43fc06b960 0x7f43fc0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440192b640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43fc0b1740 0x7f43fc1dee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440292d640 1 -- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43fc0b1740 msgr2=0x7f43fc1dee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440292d640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43fc0b1740 0x7f43fc1dee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.577+0000 7f440292d640 1 -- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43fc06b960 msgr2=0x7f43fc0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f440292d640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43fc06b960 0x7f43fc0b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f440292d640 1 -- 172.21.15.6:0/2638917355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc1e2dd0 con 0x7f43fc0b1b20 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f440212c640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43fc06b960 0x7f43fc0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f440292d640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1b20 0x7f43fc0b4f10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f43f400cc30 tx=0x7f43f400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f43f2ffd640 1 -- 172.21.15.6:0/2638917355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f400ce40 con 0x7f43fc0b1b20 2024-03-18T14:43:24.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f43f2ffd640 1 -- 172.21.15.6:0/2638917355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4003990 con 0x7f43fc0b1b20 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc1e30b0 con 0x7f43fc0b1b20 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f43f2ffd640 1 -- 172.21.15.6:0/2638917355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4003d40 con 0x7f43fc0b1b20 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.578+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc1e3670 con 0x7f43fc0b1b20 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.579+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43fc0b2270 con 0x7f43fc0b1b20 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.579+0000 7f43f2ffd640 1 -- 172.21.15.6:0/2638917355 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f43f4003ee0 con 0x7f43fc0b1b20 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.580+0000 7f43f2ffd640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f43e006bbd0 0x7f43e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.580+0000 7f43f2ffd640 1 -- 172.21.15.6:0/2638917355 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f43f408b1f0 con 0x7f43fc0b1b20 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.580+0000 7f440212c640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f43e006bbd0 0x7f43e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.580+0000 7f440212c640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f43e006bbd0 0x7f43e006e090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f43f8036000 tx=0x7f43f803b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.584+0000 7f43f2ffd640 1 -- 172.21.15.6:0/2638917355 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f43f4014030 con 0x7f43fc0b1b20 2024-03-18T14:43:24.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.806+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43fc04f620 con 0x7f43fc0b1b20 2024-03-18T14:43:24.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.807+0000 7f43f2ffd640 1 -- 172.21.15.6:0/2638917355 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5171 (secure 0 0 0) 0x7f43f4059a40 con 0x7f43fc0b1b20 2024-03-18T14:43:24.809 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:24.809 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:42:56.672813+0000","modified":"2024-03-18T14:43:04.715542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5529":{"gid":5529,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2677248951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2677248951},{"type":"v1","addr":"172.21.15.6:6834","nonce":2677248951}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5529,"qdb_cluster":[5529]},"id":2}]} 2024-03-18T14:43:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f43e006bbd0 msgr2=0x7f43e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f43e006bbd0 0x7f43e006e090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f43f8036000 tx=0x7f43f803b040 comp rx=0 tx=0).stop 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1b20 msgr2=0x7f43fc0b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1b20 0x7f43fc0b4f10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f43f400cc30 tx=0x7f43f400a5c0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 shutdown_connections 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f43e006bbd0 0x7f43e006e090 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43fc0b1b20 0x7f43fc0b4f10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43fc0b1740 0x7f43fc1dee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 --2- 172.21.15.6:0/2638917355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43fc06b960 0x7f43fc0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 >> 172.21.15.6:0/2638917355 conn(0x7f43fc0d09a0 msgr2=0x7f43fc06f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 shutdown_connections 2024-03-18T14:43:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.809+0000 7f4404390640 1 -- 172.21.15.6:0/2638917355 wait complete. 2024-03-18T14:43:24.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:24.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.922+0000 7fc5829df640 1 Processor -- start 2024-03-18T14:43:24.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.922+0000 7fc5829df640 1 -- start start 2024-03-18T14:43:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc5829df640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc57c0ced00 0x7fc57c162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc5829df640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc57c0d3210 0x7fc57c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc5829df640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc57c0b9660 con 0x7fc57c15e020 2024-03-18T14:43:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc5829df640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc57c0d2270 con 0x7fc57c0ced00 2024-03-18T14:43:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc5829df640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc57c0ce390 con 0x7fc57c0d3210 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc57c0ced00 0x7fc57c162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc57c15e020 0x7fc57c0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59792/0 (socket says 172.21.15.6:59792) 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc57c0ced00 0x7fc57c162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34738/0 (socket says 172.21.15.6:34738) 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57bfff640 1 -- 172.21.15.6:0/380948536 learned_addr learned my addr 172.21.15.6:0/380948536 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57b7fe640 1 -- 172.21.15.6:0/380948536 >> v2:172.21.15.92:3300/0 conn(0x7fc57c0d3210 msgr2=0x7fc57c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc580f7c640 1 --2- 172.21.15.6:0/380948536 >> v2:172.21.15.92:3300/0 conn(0x7fc57c0d3210 0x7fc57c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57affd640 1 -- 172.21.15.6:0/380948536 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796824929 0 0) 0x7fc57c0b9660 con 0x7fc57c15e020 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57b7fe640 1 --2- 172.21.15.6:0/380948536 >> v2:172.21.15.92:3300/0 conn(0x7fc57c0d3210 0x7fc57c0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57b7fe640 1 -- 172.21.15.6:0/380948536 >> v1:172.21.15.6:6789/0 conn(0x7fc57c15e020 legacy=0x7fc57c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.923+0000 7fc57b7fe640 1 -- 172.21.15.6:0/380948536 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57c0b9900 con 0x7fc57c0ced00 2024-03-18T14:43:24.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc57b7fe640 1 --2- 172.21.15.6:0/380948536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c162380 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc57c15e400 tx=0x7fc57002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32d90577c0c3af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc57affd640 1 -- 172.21.15.6:0/380948536 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5700315a0 con 0x7fc57c0ced00 2024-03-18T14:43:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc57affd640 1 -- 172.21.15.6:0/380948536 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc570037070 con 0x7fc57c0ced00 2024-03-18T14:43:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc57affd640 1 -- 172.21.15.6:0/380948536 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570031960 con 0x7fc57c0ced00 2024-03-18T14:43:24.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc5829df640 1 -- 172.21.15.6:0/380948536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 msgr2=0x7fc57c162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc5829df640 1 --2- 172.21.15.6:0/380948536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c162380 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc57c15e400 tx=0x7fc57002b840 comp rx=0 tx=0).stop 2024-03-18T14:43:24.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc5829df640 1 -- 172.21.15.6:0/380948536 shutdown_connections 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc5829df640 1 --2- 172.21.15.6:0/380948536 >> v2:172.21.15.92:3300/0 conn(0x7fc57c0d3210 0x7fc57c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc5829df640 1 --2- 172.21.15.6:0/380948536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c162380 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.924+0000 7fc5829df640 1 -- 172.21.15.6:0/380948536 >> 172.21.15.6:0/380948536 conn(0x7fc57c072ce0 msgr2=0x7fc57c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.925+0000 7fc5829df640 1 -- 172.21.15.6:0/380948536 shutdown_connections 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.925+0000 7fc5829df640 1 -- 172.21.15.6:0/380948536 wait complete. 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 Processor -- start 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 -- start start 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc57c0d3210 0x7fc57c15fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc57c15e020 0x7fc57c15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c0cf2c0 con 0x7fc57c0ced00 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c0ce890 con 0x7fc57c15e020 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.926+0000 7fc5829df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc57c0cf440 con 0x7fc57c0d3210 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34748/0 (socket says 172.21.15.6:34748) 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 -- 172.21.15.6:0/2847617354 learned_addr learned my addr 172.21.15.6:0/2847617354 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc580f7c640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc57c15e020 0x7fc57c15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 -- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc57c0d3210 msgr2=0x7fc57c15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57b7fe640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc57c0d3210 0x7fc57c15fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc57c0d3210 0x7fc57c15fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 -- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc57c15e020 msgr2=0x7fc57c15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:24.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc57c15e020 0x7fc57c15ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 -- 172.21.15.6:0/2847617354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57c0b4010 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc580f7c640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc57c15e020 0x7fc57c15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc57bfff640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c157fc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc56c00c400 tx=0x7fc56c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc578ff9640 1 -- 172.21.15.6:0/2847617354 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c017070 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc578ff9640 1 -- 172.21.15.6:0/2847617354 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56c005ce0 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc57c0b42f0 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc578ff9640 1 -- 172.21.15.6:0/2847617354 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c007310 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.927+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc57c0b48b0 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.929+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc57c08afe0 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.929+0000 7fc578ff9640 1 -- 172.21.15.6:0/2847617354 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc56c01a070 con 0x7fc57c0ced00 2024-03-18T14:43:24.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.929+0000 7fc578ff9640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc56806bbd0 0x7fc56806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:24.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.929+0000 7fc578ff9640 1 -- 172.21.15.6:0/2847617354 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7fc56c08a290 con 0x7fc57c0ced00 2024-03-18T14:43:24.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.929+0000 7fc57b7fe640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc56806bbd0 0x7fc56806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:24.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.930+0000 7fc57b7fe640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc56806bbd0 0x7fc56806e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc570036000 tx=0x7fc570008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:24.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:24.934+0000 7fc578ff9640 1 -- 172.21.15.6:0/2847617354 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc56c0589c0 con 0x7fc57c0ced00 2024-03-18T14:43:25.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.141+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc57c06e070 con 0x7fc57c0ced00 2024-03-18T14:43:25.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.143+0000 7fc578ff9640 1 -- 172.21.15.6:0/2847617354 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+18707 (secure 0 0 0) 0x7fc56c0582c0 con 0x7fc57c0ced00 2024-03-18T14:43:25.144 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:25.145 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:17.388447+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc56806bbd0 msgr2=0x7fc56806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc56806bbd0 0x7fc56806e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc570036000 tx=0x7fc570008030 comp rx=0 tx=0).stop 2024-03-18T14:43:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 msgr2=0x7fc57c157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c157fc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc56c00c400 tx=0x7fc56c00c8c0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 shutdown_connections 2024-03-18T14:43:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc56806bbd0 0x7fc56806e090 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc57c15e020 0x7fc57c15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc57c0d3210 0x7fc57c15fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 --2- 172.21.15.6:0/2847617354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc57c0ced00 0x7fc57c157fc0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 >> 172.21.15.6:0/2847617354 conn(0x7fc57c072ce0 msgr2=0x7fc57c0d2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.145+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 shutdown_connections 2024-03-18T14:43:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.146+0000 7fc5829df640 1 -- 172.21.15.6:0/2847617354 wait complete. 2024-03-18T14:43:25.158 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:42:54.939648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:42:55.638120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-18T14:43:03.985735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:25.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:25.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.265+0000 7f404cf62640 1 Processor -- start 2024-03-18T14:43:25.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.265+0000 7f404cf62640 1 -- start start 2024-03-18T14:43:25.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f404cf62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f404cf62640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f40480b6140 0x7f40480ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f404cf62640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40480b65a0 con 0x7f40480d0450 2024-03-18T14:43:25.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f404cf62640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f404815e410 con 0x7f40480b6140 2024-03-18T14:43:25.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f404cf62640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f40480ce300 con 0x7f404815e030 2024-03-18T14:43:25.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f4046d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f40480d0450 0x7f40480d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59796/0 (socket says 172.21.15.6:59796) 2024-03-18T14:43:25.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f4046d9d640 1 -- 172.21.15.6:0/1805015833 learned_addr learned my addr 172.21.15.6:0/1805015833 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f403ffff640 1 --2- 172.21.15.6:0/1805015833 >> v2:172.21.15.92:3300/0 conn(0x7f40480b6140 0x7f40480ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f404659c640 1 --2- 172.21.15.6:0/1805015833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f4045d9b640 1 -- 172.21.15.6:0/1805015833 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 467034152 0 0) 0x7f40480b65a0 con 0x7f40480d0450 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.266+0000 7f4045d9b640 1 -- 172.21.15.6:0/1805015833 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4034003520 con 0x7f40480d0450 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.267+0000 7f404659c640 1 -- 172.21.15.6:0/1805015833 >> v2:172.21.15.92:3300/0 conn(0x7f40480b6140 msgr2=0x7f40480ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.267+0000 7f404659c640 1 --2- 172.21.15.6:0/1805015833 >> v2:172.21.15.92:3300/0 conn(0x7f40480b6140 0x7f40480ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.267+0000 7f404659c640 1 -- 172.21.15.6:0/1805015833 >> v1:172.21.15.6:6789/0 conn(0x7f40480d0450 legacy=0x7f40480d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.267+0000 7f404659c640 1 -- 172.21.15.6:0/1805015833 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40480d2fd0 con 0x7f404815e030 2024-03-18T14:43:25.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.267+0000 7f403ffff640 1 --2- 172.21.15.6:0/1805015833 >> v2:172.21.15.92:3300/0 conn(0x7f40480b6140 0x7f40480ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.267+0000 7f404659c640 1 --2- 172.21.15.6:0/1805015833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480d3570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f40380097a0 tx=0x7f403802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f716754a6a86d69f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f4045d9b640 1 -- 172.21.15.6:0/1805015833 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4038035020 con 0x7f404815e030 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f4045d9b640 1 -- 172.21.15.6:0/1805015833 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4038031830 con 0x7f404815e030 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f4045d9b640 1 -- 172.21.15.6:0/1805015833 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4038031c00 con 0x7f404815e030 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 -- 172.21.15.6:0/1805015833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 msgr2=0x7f40480d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 --2- 172.21.15.6:0/1805015833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480d3570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f40380097a0 tx=0x7f403802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 -- 172.21.15.6:0/1805015833 shutdown_connections 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 --2- 172.21.15.6:0/1805015833 >> v2:172.21.15.92:3300/0 conn(0x7f40480b6140 0x7f40480ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 --2- 172.21.15.6:0/1805015833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480d3570 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 -- 172.21.15.6:0/1805015833 >> 172.21.15.6:0/1805015833 conn(0x7f404806c8c0 msgr2=0x7f404806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:25.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 -- 172.21.15.6:0/1805015833 shutdown_connections 2024-03-18T14:43:25.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.268+0000 7f404cf62640 1 -- 172.21.15.6:0/1805015833 wait complete. 2024-03-18T14:43:25.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.270+0000 7f404cf62640 1 Processor -- start 2024-03-18T14:43:25.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.270+0000 7f404cf62640 1 -- start start 2024-03-18T14:43:25.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404cf62640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40480b6140 0x7f404815e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404cf62640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40480d0450 0x7f404815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404cf62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404cf62640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f40480ce830 con 0x7f40480b6140 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404cf62640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40480ce550 con 0x7f40480d0450 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404cf62640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f40480cf400 con 0x7f404815e030 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40480b6140 0x7f404815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40480b6140 0x7f404815e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34754/0 (socket says 172.21.15.6:34754) 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404659c640 1 -- 172.21.15.6:0/656047602 learned_addr learned my addr 172.21.15.6:0/656047602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f403ffff640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40480d0450 0x7f404815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f4046d9d640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404659c640 1 -- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 msgr2=0x7f40480cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404659c640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404659c640 1 -- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40480d0450 msgr2=0x7f404815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.271+0000 7f404659c640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40480d0450 0x7f404815a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.272+0000 7f404659c640 1 -- 172.21.15.6:0/656047602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40480cc5f0 con 0x7f40480b6140 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.272+0000 7f404659c640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40480b6140 0x7f404815e590 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f403802bee0 tx=0x7f40380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.272+0000 7f403effd640 1 -- 172.21.15.6:0/656047602 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4038035040 con 0x7f40480b6140 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.272+0000 7f403effd640 1 -- 172.21.15.6:0/656047602 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f403803b920 con 0x7f40480b6140 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.272+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40480cc8d0 con 0x7f40480b6140 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.272+0000 7f403effd640 1 -- 172.21.15.6:0/656047602 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403803bd10 con 0x7f40480b6140 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.272+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40480cce90 con 0x7f40480b6140 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.273+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40480751b0 con 0x7f40480b6140 2024-03-18T14:43:25.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.273+0000 7f403effd640 1 -- 172.21.15.6:0/656047602 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4038038020 con 0x7f40480b6140 2024-03-18T14:43:25.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.274+0000 7f403effd640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f402806bc00 0x7f402806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.274+0000 7f403effd640 1 -- 172.21.15.6:0/656047602 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f40380acd10 con 0x7f40480b6140 2024-03-18T14:43:25.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.274+0000 7f403ffff640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f402806bc00 0x7f402806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.274+0000 7f403ffff640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f402806bc00 0x7f402806e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4030009560 tx=0x7f4030009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.278+0000 7f403effd640 1 -- 172.21.15.6:0/656047602 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f403807af10 con 0x7f40480b6140 2024-03-18T14:43:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.464+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f40480d21b0 con 0x7f40480b6140 2024-03-18T14:43:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.465+0000 7f403effd640 1 -- 172.21.15.6:0/656047602 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7f403807efb0 con 0x7f40480b6140 2024-03-18T14:43:25.466 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:25.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.466+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f402806bc00 msgr2=0x7f402806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.466+0000 7f404cf62640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f402806bc00 0x7f402806e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4030009560 tx=0x7f4030009190 comp rx=0 tx=0).stop 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.466+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40480b6140 msgr2=0x7f404815e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.466+0000 7f404cf62640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40480b6140 0x7f404815e590 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f403802bee0 tx=0x7f40380046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 shutdown_connections 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f402806bc00 0x7f402806e0c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404815e030 0x7f40480cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40480d0450 0x7f404815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 --2- 172.21.15.6:0/656047602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40480b6140 0x7f404815e590 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 >> 172.21.15.6:0/656047602 conn(0x7f404806c8c0 msgr2=0x7f40480d43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 shutdown_connections 2024-03-18T14:43:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.467+0000 7f404cf62640 1 -- 172.21.15.6:0/656047602 wait complete. 2024-03-18T14:43:25.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.586+0000 7fa8ba8e8640 1 Processor -- start 2024-03-18T14:43:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.586+0000 7fa8ba8e8640 1 -- start start 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8ba8e8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8b4161740 0x7fa8b409ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8ba8e8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b40b4920 con 0x7fa8b409a3c0 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8ba8e8640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b406d740 con 0x7fa8b406cef0 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8ba8e8640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa8b40cc120 con 0x7fa8b4161740 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8b3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa8b4161740 0x7fa8b409ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8b8e85640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8b409a3c0 0x7fa8b40cce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59812/0 (socket says 172.21.15.6:59812) 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8b8e85640 1 -- 172.21.15.6:0/2625515660 learned_addr learned my addr 172.21.15.6:0/2625515660 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8b3fff640 1 -- 172.21.15.6:0/2625515660 >> v1:172.21.15.92:6789/0 conn(0x7fa8b406cef0 legacy=0x7fa8b4099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8b3fff640 1 -- 172.21.15.6:0/2625515660 >> v1:172.21.15.6:6789/0 conn(0x7fa8b409a3c0 legacy=0x7fa8b40cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8b3fff640 1 -- 172.21.15.6:0/2625515660 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b40cc370 con 0x7fa8b4161740 2024-03-18T14:43:25.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.587+0000 7fa8b2ffd640 1 -- 172.21.15.6:0/2625515660 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914031044 0 0) 0x7fa8ac001490 con 0x7fa8b409a3c0 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.588+0000 7fa8b3fff640 1 --2- 172.21.15.6:0/2625515660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b409ad30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939596abb559f4a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.588+0000 7fa8b2ffd640 1 -- 172.21.15.6:0/2625515660 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8035020 con 0x7fa8b4161740 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.588+0000 7fa8b2ffd640 1 -- 172.21.15.6:0/2625515660 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a8031830 con 0x7fa8b4161740 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.588+0000 7fa8b2ffd640 1 -- 172.21.15.6:0/2625515660 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8031c00 con 0x7fa8b4161740 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.589+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/2625515660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 msgr2=0x7fa8b409ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.589+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/2625515660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b409ad30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.589+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/2625515660 shutdown_connections 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.589+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/2625515660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b409ad30 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.589+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/2625515660 >> 172.21.15.6:0/2625515660 conn(0x7fa8b40b35c0 msgr2=0x7fa8b40b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.589+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/2625515660 shutdown_connections 2024-03-18T14:43:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.589+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/2625515660 wait complete. 2024-03-18T14:43:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.590+0000 7fa8ba8e8640 1 Processor -- start 2024-03-18T14:43:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.590+0000 7fa8ba8e8640 1 -- start start 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8ba8e8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8b406cef0 0x7fa8b41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8ba8e8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8b409a3c0 0x7fa8b40b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8ba8e8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8ba8e8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b406d330 con 0x7fa8b4161740 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8ba8e8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b40cc930 con 0x7fa8b409a3c0 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8ba8e8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b406d4b0 con 0x7fa8b406cef0 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b415fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34766/0 (socket says 172.21.15.6:34766) 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 -- 172.21.15.6:0/3583663554 learned_addr learned my addr 172.21.15.6:0/3583663554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b37fe640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8b409a3c0 0x7fa8b40b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b3fff640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8b406cef0 0x7fa8b41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 -- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8b406cef0 msgr2=0x7fa8b41563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8b406cef0 0x7fa8b41563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 -- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8b409a3c0 msgr2=0x7fa8b40b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8b409a3c0 0x7fa8b40b7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 -- 172.21.15.6:0/3583663554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b40b61c0 con 0x7fa8b4161740 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b8e85640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b415fdb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00c7c0 tx=0x7fa8ac00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b0ff9640 1 -- 172.21.15.6:0/3583663554 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac018070 con 0x7fa8b4161740 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b0ff9640 1 -- 172.21.15.6:0/3583663554 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8ac00a5e0 con 0x7fa8b4161740 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b40b64a0 con 0x7fa8b4161740 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.591+0000 7fa8b0ff9640 1 -- 172.21.15.6:0/3583663554 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00a9d0 con 0x7fa8b4161740 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.592+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b40b6a60 con 0x7fa8b4161740 2024-03-18T14:43:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.592+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b40751b0 con 0x7fa8b4161740 2024-03-18T14:43:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.593+0000 7fa8b0ff9640 1 -- 172.21.15.6:0/3583663554 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa8ac01b070 con 0x7fa8b4161740 2024-03-18T14:43:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.593+0000 7fa8b0ff9640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa8a006bbd0 0x7fa8a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.593+0000 7fa8b0ff9640 1 -- 172.21.15.6:0/3583663554 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7fa8ac08a2a0 con 0x7fa8b4161740 2024-03-18T14:43:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.594+0000 7fa8b3fff640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa8a006bbd0 0x7fa8a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.594+0000 7fa8b3fff640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa8a006bbd0 0x7fa8a006e090 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8036000 tx=0x7fa8a803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.602+0000 7fa8b0ff9640 1 -- 172.21.15.6:0/3583663554 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa8b40b4920 con 0x7fa8b4161740 2024-03-18T14:43:25.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.823+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa8b406eca0 con 0x7fa8b4161740 2024-03-18T14:43:25.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.824+0000 7fa8b0ff9640 1 -- 172.21.15.6:0/3583663554 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7fa8ac0584a0 con 0x7fa8b4161740 2024-03-18T14:43:25.825 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa8a006bbd0 msgr2=0x7fa8a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa8a006bbd0 0x7fa8a006e090 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8036000 tx=0x7fa8a803d040 comp rx=0 tx=0).stop 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 msgr2=0x7fa8b415fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b415fdb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00c7c0 tx=0x7fa8ac00cc80 comp rx=0 tx=0).stop 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 shutdown_connections 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa8a006bbd0 0x7fa8a006e090 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b4161740 0x7fa8b415fdb0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8b409a3c0 0x7fa8b40b7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 --2- 172.21.15.6:0/3583663554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8b406cef0 0x7fa8b41563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 >> 172.21.15.6:0/3583663554 conn(0x7fa8b40b35c0 msgr2=0x7fa8b406c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:25.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.826+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 shutdown_connections 2024-03-18T14:43:25.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.827+0000 7fa8ba8e8640 1 -- 172.21.15.6:0/3583663554 wait complete. 2024-03-18T14:43:25.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:25.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.945+0000 7f59c6787640 1 Processor -- start 2024-03-18T14:43:25.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.945+0000 7f59c6787640 1 -- start start 2024-03-18T14:43:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59c6787640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f59c00d3570 0x7f59c00b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59c6787640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c00d39d0 con 0x7f59c00d0450 2024-03-18T14:43:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59c6787640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c00d2fd0 con 0x7f59c015e030 2024-03-18T14:43:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59c6787640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f59c015e410 con 0x7f59c00d3570 2024-03-18T14:43:25.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59c4d24640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f59c00d0450 0x7f59c00d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59822/0 (socket says 172.21.15.6:59822) 2024-03-18T14:43:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59c4d24640 1 -- 172.21.15.6:0/694941584 learned_addr learned my addr 172.21.15.6:0/694941584 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59bf7fe640 1 --2- 172.21.15.6:0/694941584 >> v2:172.21.15.92:3300/0 conn(0x7f59c00d3570 0x7f59c00b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59beffd640 1 -- 172.21.15.6:0/694941584 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640653995 0 0) 0x7f59c00d39d0 con 0x7f59c00d0450 2024-03-18T14:43:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.946+0000 7f59beffd640 1 -- 172.21.15.6:0/694941584 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a8003520 con 0x7f59c00d0450 2024-03-18T14:43:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.947+0000 7f59bf7fe640 1 -- 172.21.15.6:0/694941584 >> v1:172.21.15.92:6789/0 conn(0x7f59c015e030 legacy=0x7f59c00ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.947+0000 7f59bf7fe640 1 -- 172.21.15.6:0/694941584 >> v1:172.21.15.6:6789/0 conn(0x7f59c00d0450 legacy=0x7f59c00d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.947+0000 7f59bf7fe640 1 -- 172.21.15.6:0/694941584 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c00d3290 con 0x7f59c00d3570 2024-03-18T14:43:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.947+0000 7f59bf7fe640 1 --2- 172.21.15.6:0/694941584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c00d3570 0x7f59c00b6140 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f59b40097a0 tx=0x7f59b402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b81c416aada773f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59beffd640 1 -- 172.21.15.6:0/694941584 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4034070 con 0x7f59c00d3570 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59beffd640 1 -- 172.21.15.6:0/694941584 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b4030610 con 0x7f59c00d3570 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59beffd640 1 -- 172.21.15.6:0/694941584 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b40309e0 con 0x7f59c00d3570 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59c6787640 1 -- 172.21.15.6:0/694941584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c00d3570 msgr2=0x7f59c00b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59c6787640 1 --2- 172.21.15.6:0/694941584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c00d3570 0x7f59c00b6140 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f59b40097a0 tx=0x7f59b402b960 comp rx=0 tx=0).stop 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59c6787640 1 -- 172.21.15.6:0/694941584 shutdown_connections 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59c6787640 1 --2- 172.21.15.6:0/694941584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c00d3570 0x7f59c00b6140 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59c6787640 1 -- 172.21.15.6:0/694941584 >> 172.21.15.6:0/694941584 conn(0x7f59c006c8c0 msgr2=0x7f59c006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59c6787640 1 -- 172.21.15.6:0/694941584 shutdown_connections 2024-03-18T14:43:25.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.948+0000 7f59c6787640 1 -- 172.21.15.6:0/694941584 wait complete. 2024-03-18T14:43:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.950+0000 7f59c6787640 1 Processor -- start 2024-03-18T14:43:25.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.950+0000 7f59c6787640 1 -- start start 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59c6787640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59c00d0450 0x7f59c00cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59c6787640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59c00d3570 0x7f59c00b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59c6787640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c015e030 0x7f59c0154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59c6787640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f59c00ce830 con 0x7f59c00d0450 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59c6787640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f59c00ce300 con 0x7f59c015e030 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59c6787640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f59c0161a90 con 0x7f59c00d3570 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59bffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59c00d0450 0x7f59c00cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59bffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59c00d0450 0x7f59c00cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34774/0 (socket says 172.21.15.6:34774) 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.951+0000 7f59bffff640 1 -- 172.21.15.6:0/2591327196 learned_addr learned my addr 172.21.15.6:0/2591327196 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:25.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59c4d24640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c015e030 0x7f59c0154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bf7fe640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59c00d3570 0x7f59c00b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bffff640 1 -- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59c00d3570 msgr2=0x7f59c00b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bffff640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59c00d3570 0x7f59c00b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bffff640 1 -- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c015e030 msgr2=0x7f59c0154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bffff640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c015e030 0x7f59c0154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bffff640 1 -- 172.21.15.6:0/2591327196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c01e8e20 con 0x7f59c00d0450 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59c4d24640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c015e030 0x7f59c0154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bffff640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59c00d0450 0x7f59c00cb790 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c3c0 tx=0x7f59b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bcff9640 1 -- 172.21.15.6:0/2591327196 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b00052d0 con 0x7f59c00d0450 2024-03-18T14:43:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bcff9640 1 -- 172.21.15.6:0/2591327196 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b0005490 con 0x7f59c00d0450 2024-03-18T14:43:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c01e9100 con 0x7f59c00d0450 2024-03-18T14:43:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59bcff9640 1 -- 172.21.15.6:0/2591327196 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b0007260 con 0x7f59c00d0450 2024-03-18T14:43:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.952+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c01e96c0 con 0x7f59c00d0450 2024-03-18T14:43:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.953+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c01e9af0 con 0x7f59c00d0450 2024-03-18T14:43:25.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.954+0000 7f59bcff9640 1 -- 172.21.15.6:0/2591327196 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f59b0007400 con 0x7f59c00d0450 2024-03-18T14:43:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.954+0000 7f59bcff9640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f59ac06bbd0 0x7f59ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.954+0000 7f59bcff9640 1 -- 172.21.15.6:0/2591327196 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f59b008a280 con 0x7f59c00d0450 2024-03-18T14:43:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.954+0000 7f59bf7fe640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f59ac06bbd0 0x7f59ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:25.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.955+0000 7f59bf7fe640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f59ac06bbd0 0x7f59ac06e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f59b4036000 tx=0x7f59b403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:25.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:25.958+0000 7f59bcff9640 1 -- 172.21.15.6:0/2591327196 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f59b0058a40 con 0x7f59c00d0450 2024-03-18T14:43:26.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.137+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f59c00d19f0 con 0x7f59c00d0450 2024-03-18T14:43:26.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.138+0000 7f59bcff9640 1 -- 172.21.15.6:0/2591327196 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7f59b0058340 con 0x7f59c00d0450 2024-03-18T14:43:26.139 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f59ac06bbd0 msgr2=0x7f59ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f59ac06bbd0 0x7f59ac06e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f59b4036000 tx=0x7f59b403b040 comp rx=0 tx=0).stop 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59c00d0450 msgr2=0x7f59c00cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59c00d0450 0x7f59c00cb790 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c3c0 tx=0x7f59b000c880 comp rx=0 tx=0).stop 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 shutdown_connections 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f59ac06bbd0 0x7f59ac06e090 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59c015e030 0x7f59c0154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59c00d3570 0x7f59c00b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 --2- 172.21.15.6:0/2591327196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59c00d0450 0x7f59c00cb790 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 >> 172.21.15.6:0/2591327196 conn(0x7f59c006c8c0 msgr2=0x7f59c015ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 shutdown_connections 2024-03-18T14:43:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.140+0000 7f59c6787640 1 -- 172.21.15.6:0/2591327196 wait complete. 2024-03-18T14:43:26.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-18T14:43:26.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.281+0000 7f7405d87640 1 Processor -- start 2024-03-18T14:43:26.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.281+0000 7f7405d87640 1 -- start start 2024-03-18T14:43:26.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f7405d87640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f740015e030 0x7f74000d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f7405d87640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74000b6140 0x7f74000ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f7405d87640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74000b65a0 con 0x7f74000d0450 2024-03-18T14:43:26.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f7405d87640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f740015e410 con 0x7f74000b6140 2024-03-18T14:43:26.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f7405d87640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f74000ce300 con 0x7f740015e030 2024-03-18T14:43:26.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73feffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74000b6140 0x7f74000ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73feffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74000b6140 0x7f74000ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34788/0 (socket says 172.21.15.6:34788) 2024-03-18T14:43:26.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73feffd640 1 -- 172.21.15.6:0/19332771 learned_addr learned my addr 172.21.15.6:0/19332771 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:26.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73ff7fe640 1 --2- 172.21.15.6:0/19332771 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f740015e030 0x7f74000d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73feffd640 1 -- 172.21.15.6:0/19332771 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f740015e030 msgr2=0x7f74000d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:26.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73feffd640 1 --2- 172.21.15.6:0/19332771 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f740015e030 0x7f74000d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73feffd640 1 -- 172.21.15.6:0/19332771 >> v1:172.21.15.92:6789/0 conn(0x7f74000d0450 legacy=0x7f74000d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.282+0000 7f73feffd640 1 -- 172.21.15.6:0/19332771 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74000d2fd0 con 0x7f74000b6140 2024-03-18T14:43:26.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.283+0000 7f73feffd640 1 --2- 172.21.15.6:0/19332771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000ceec0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009870 tx=0x7f73f002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560edade7a4c7eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:26.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f73fe7fc640 1 -- 172.21.15.6:0/19332771 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0034070 con 0x7f74000b6140 2024-03-18T14:43:26.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f73fe7fc640 1 -- 172.21.15.6:0/19332771 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f0030840 con 0x7f74000b6140 2024-03-18T14:43:26.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f73fe7fc640 1 -- 172.21.15.6:0/19332771 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0030c10 con 0x7f74000b6140 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 -- 172.21.15.6:0/19332771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 msgr2=0x7f74000ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 --2- 172.21.15.6:0/19332771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000ceec0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009870 tx=0x7f73f002b980 comp rx=0 tx=0).stop 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 -- 172.21.15.6:0/19332771 shutdown_connections 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 --2- 172.21.15.6:0/19332771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000ceec0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 --2- 172.21.15.6:0/19332771 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f740015e030 0x7f74000d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 -- 172.21.15.6:0/19332771 >> 172.21.15.6:0/19332771 conn(0x7f740006c8c0 msgr2=0x7f740006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 -- 172.21.15.6:0/19332771 shutdown_connections 2024-03-18T14:43:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.284+0000 7f7405d87640 1 -- 172.21.15.6:0/19332771 wait complete. 2024-03-18T14:43:26.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.286+0000 7f7405d87640 1 Processor -- start 2024-03-18T14:43:26.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.286+0000 7f7405d87640 1 -- start start 2024-03-18T14:43:26.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.287+0000 7f7405d87640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.287+0000 7f7405d87640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000d0450 0x7f74000cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f7405d87640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f740015e030 0x7f740015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f7405d87640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f74000ce830 con 0x7f74000b6140 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f7405d87640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74000ce550 con 0x7f740015e030 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f7405d87640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f74000cf400 con 0x7f74000d0450 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34792/0 (socket says 172.21.15.6:34792) 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 -- 172.21.15.6:0/3585266316 learned_addr learned my addr 172.21.15.6:0/3585266316 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73feffd640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000d0450 0x7f74000cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 -- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000d0450 msgr2=0x7f74000cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73fffff640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f740015e030 0x7f740015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000d0450 0x7f74000cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 -- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f740015e030 msgr2=0x7f740015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f740015e030 0x7f740015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 -- 172.21.15.6:0/3585266316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74000cce00 con 0x7f74000b6140 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73feffd640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000d0450 0x7f74000cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73fffff640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f740015e030 0x7f740015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73ff7fe640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000cb950 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f73f400d800 tx=0x7f73f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73e7fff640 1 -- 172.21.15.6:0/3585266316 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f400b6c0 con 0x7f74000b6140 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73e7fff640 1 -- 172.21.15.6:0/3585266316 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f4019070 con 0x7f74000b6140 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74000cd0e0 con 0x7f74000b6140 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f73e7fff640 1 -- 172.21.15.6:0/3585266316 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f400ba80 con 0x7f74000b6140 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.288+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74000cd6a0 con 0x7f74000b6140 2024-03-18T14:43:26.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.289+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74000751b0 con 0x7f74000b6140 2024-03-18T14:43:26.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.290+0000 7f73e7fff640 1 -- 172.21.15.6:0/3585266316 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f73f400bc20 con 0x7f74000b6140 2024-03-18T14:43:26.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.290+0000 7f73e7fff640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73ec06bbd0 0x7f73ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.290+0000 7f73e7fff640 1 -- 172.21.15.6:0/3585266316 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f73f408b200 con 0x7f74000b6140 2024-03-18T14:43:26.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.290+0000 7f73feffd640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73ec06bbd0 0x7f73ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.291+0000 7f73feffd640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73ec06bbd0 0x7f73ec06e090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f73f0036000 tx=0x7f73f003b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:26.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.295+0000 7f73e7fff640 1 -- 172.21.15.6:0/3585266316 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f73f4059a50 con 0x7f74000b6140 2024-03-18T14:43:26.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.564+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f74000b9970 con 0x7f74000b6140 2024-03-18T14:43:26.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.564+0000 7f73e7fff640 1 -- 172.21.15.6:0/3585266316 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v44) ==== 97+0+11 (secure 0 0 0) 0x7f73f4059350 con 0x7f74000b6140 2024-03-18T14:43:26.566 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:26.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73ec06bbd0 msgr2=0x7f73ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73ec06bbd0 0x7f73ec06e090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f73f0036000 tx=0x7f73f003b040 comp rx=0 tx=0).stop 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 msgr2=0x7f74000cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000cb950 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f73f400d800 tx=0x7f73f400dcc0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 shutdown_connections 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73ec06bbd0 0x7f73ec06e090 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f740015e030 0x7f740015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000d0450 0x7f74000cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 --2- 172.21.15.6:0/3585266316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74000b6140 0x7f74000cb950 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 >> 172.21.15.6:0/3585266316 conn(0x7f740006c8c0 msgr2=0x7f740006cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 shutdown_connections 2024-03-18T14:43:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.566+0000 7f7405d87640 1 -- 172.21.15.6:0/3585266316 wait complete. 2024-03-18T14:43:26.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.680+0000 7f20353b2640 1 Processor -- start 2024-03-18T14:43:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.680+0000 7f20353b2640 1 -- start start 2024-03-18T14:43:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f20353b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300ced00 0x7f2030162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f20353b2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20300d3210 0x7f20300d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f20353b2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20300b9660 con 0x7f203015e020 2024-03-18T14:43:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f20353b2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f20300d2270 con 0x7f20300d3210 2024-03-18T14:43:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f20353b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f20300ce390 con 0x7f20300ced00 2024-03-18T14:43:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f202f7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20300d3210 0x7f20300d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f202e7fc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300ced00 0x7f2030162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f202effd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f203015e020 0x7f20300d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45846/0 (socket says 172.21.15.6:45846) 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.681+0000 7f202effd640 1 -- 172.21.15.6:0/2513997702 learned_addr learned my addr 172.21.15.6:0/2513997702 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202dffb640 1 -- 172.21.15.6:0/2513997702 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1579445956 0 0) 0x7f20300b9660 con 0x7f203015e020 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202dffb640 1 -- 172.21.15.6:0/2513997702 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f201c003520 con 0x7f203015e020 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202e7fc640 1 -- 172.21.15.6:0/2513997702 >> v2:172.21.15.92:3300/0 conn(0x7f20300d3210 msgr2=0x7f20300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202e7fc640 1 --2- 172.21.15.6:0/2513997702 >> v2:172.21.15.92:3300/0 conn(0x7f20300d3210 0x7f20300d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202e7fc640 1 -- 172.21.15.6:0/2513997702 >> v1:172.21.15.92:6789/0 conn(0x7f203015e020 legacy=0x7f20300d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202e7fc640 1 -- 172.21.15.6:0/2513997702 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20300b9900 con 0x7f20300ced00 2024-03-18T14:43:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202f7fe640 1 --2- 172.21.15.6:0/2513997702 >> v2:172.21.15.92:3300/0 conn(0x7f20300d3210 0x7f20300d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.682+0000 7f202e7fc640 1 --2- 172.21.15.6:0/2513997702 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300ced00 0x7f2030162380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f20300d23f0 tx=0x7f201802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ee201f99458e831 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f202dffb640 1 -- 172.21.15.6:0/2513997702 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018031770 con 0x7f20300ced00 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f202dffb640 1 -- 172.21.15.6:0/2513997702 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2018037070 con 0x7f20300ced00 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f202dffb640 1 -- 172.21.15.6:0/2513997702 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018031b10 con 0x7f20300ced00 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 -- 172.21.15.6:0/2513997702 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300ced00 msgr2=0x7f2030162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 --2- 172.21.15.6:0/2513997702 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300ced00 0x7f2030162380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f20300d23f0 tx=0x7f201802b810 comp rx=0 tx=0).stop 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 -- 172.21.15.6:0/2513997702 shutdown_connections 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 --2- 172.21.15.6:0/2513997702 >> v2:172.21.15.92:3300/0 conn(0x7f20300d3210 0x7f20300d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 --2- 172.21.15.6:0/2513997702 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300ced00 0x7f2030162380 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 -- 172.21.15.6:0/2513997702 >> 172.21.15.6:0/2513997702 conn(0x7f2030072ce0 msgr2=0x7f20300730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 -- 172.21.15.6:0/2513997702 shutdown_connections 2024-03-18T14:43:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.683+0000 7f20353b2640 1 -- 172.21.15.6:0/2513997702 wait complete. 2024-03-18T14:43:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.685+0000 7f20353b2640 1 Processor -- start 2024-03-18T14:43:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.685+0000 7f20353b2640 1 -- start start 2024-03-18T14:43:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20353b2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20300ced00 0x7f2030155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20353b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300d3210 0x7f2030151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20353b2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f203015e020 0x7f20300b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20353b2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f20300cf2c0 con 0x7f20300ced00 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20353b2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f20300ce890 con 0x7f203015e020 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20353b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f20300cf440 con 0x7f20300d3210 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20300ced00 0x7f2030155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20300ced00 0x7f2030155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34808/0 (socket says 172.21.15.6:34808) 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 -- 172.21.15.6:0/2803807459 learned_addr learned my addr 172.21.15.6:0/2803807459 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202e7fc640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300d3210 0x7f2030151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202f7fe640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f203015e020 0x7f20300b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 -- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300d3210 msgr2=0x7f2030151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300d3210 0x7f2030151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 -- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f203015e020 msgr2=0x7f20300b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f203015e020 0x7f20300b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 -- 172.21.15.6:0/2803807459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203015e900 con 0x7f20300ced00 2024-03-18T14:43:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f202effd640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20300ced00 0x7f2030155bf0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f202000cad0 tx=0x7f202000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20277fe640 1 -- 172.21.15.6:0/2803807459 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020017070 con 0x7f20300ced00 2024-03-18T14:43:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20277fe640 1 -- 172.21.15.6:0/2803807459 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202000ac30 con 0x7f20300ced00 2024-03-18T14:43:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f203015ebe0 con 0x7f20300ced00 2024-03-18T14:43:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.687+0000 7f20277fe640 1 -- 172.21.15.6:0/2803807459 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020012400 con 0x7f20300ced00 2024-03-18T14:43:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.688+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f203015f1a0 con 0x7f20300ced00 2024-03-18T14:43:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.688+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f203008afe0 con 0x7f20300ced00 2024-03-18T14:43:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.689+0000 7f20277fe640 1 -- 172.21.15.6:0/2803807459 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f202001a070 con 0x7f20300ced00 2024-03-18T14:43:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.689+0000 7f20277fe640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f201006bbd0 0x7f201006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.689+0000 7f20277fe640 1 -- 172.21.15.6:0/2803807459 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f202008aae0 con 0x7f20300ced00 2024-03-18T14:43:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.690+0000 7f202e7fc640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f201006bbd0 0x7f201006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.690+0000 7f202e7fc640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f201006bbd0 0x7f201006e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2018036000 tx=0x7f201803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.694+0000 7f20277fe640 1 -- 172.21.15.6:0/2803807459 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2020059330 con 0x7f20300ced00 2024-03-18T14:43:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.913+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20300b33e0 con 0x7f20300ced00 2024-03-18T14:43:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.913+0000 7f20277fe640 1 -- 172.21.15.6:0/2803807459 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5171 (secure 0 0 0) 0x7f2020058c30 con 0x7f20300ced00 2024-03-18T14:43:26.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:26.915 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:42:56.672813+0000","modified":"2024-03-18T14:43:04.715542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5529":{"gid":5529,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2677248951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2677248951},{"type":"v1","addr":"172.21.15.6:6834","nonce":2677248951}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5529,"qdb_cluster":[5529]},"id":2}]} 2024-03-18T14:43:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-18T14:43:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f201006bbd0 msgr2=0x7f201006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f201006bbd0 0x7f201006e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2018036000 tx=0x7f201803d040 comp rx=0 tx=0).stop 2024-03-18T14:43:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20300ced00 msgr2=0x7f2030155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20300ced00 0x7f2030155bf0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f202000cad0 tx=0x7f202000cf90 comp rx=0 tx=0).stop 2024-03-18T14:43:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 shutdown_connections 2024-03-18T14:43:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f201006bbd0 0x7f201006e090 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f203015e020 0x7f20300b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20300d3210 0x7f2030151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 --2- 172.21.15.6:0/2803807459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20300ced00 0x7f2030155bf0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 >> 172.21.15.6:0/2803807459 conn(0x7f2030072ce0 msgr2=0x7f20300d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.915+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 shutdown_connections 2024-03-18T14:43:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:26.916+0000 7f20353b2640 1 -- 172.21.15.6:0/2803807459 wait complete. 2024-03-18T14:43:26.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:27.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.031+0000 7f8eb609a640 1 Processor -- start 2024-03-18T14:43:27.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.032+0000 7f8eb609a640 1 -- start start 2024-03-18T14:43:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eb609a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00d1f60 0x7f8eb00cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eb609a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb00cc870 con 0x7f8eb00cfcd0 2024-03-18T14:43:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eb609a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb00d0500 con 0x7f8eb015bd40 2024-03-18T14:43:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eb609a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb00d07e0 con 0x7f8eb00d1f60 2024-03-18T14:43:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eaffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8eb00cfcd0 0x7f8eb00cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59826/0 (socket says 172.21.15.6:59826) 2024-03-18T14:43:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eaffff640 1 -- 172.21.15.6:0/2517184698 learned_addr learned my addr 172.21.15.6:0/2517184698 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8ea7fff640 1 --2- 172.21.15.6:0/2517184698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00d1f60 0x7f8eb00cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946581562 0 0) 0x7f8eb00cc870 con 0x7f8eb00cfcd0 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.033+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e9c003520 con 0x7f8eb00cfcd0 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.034+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755019593 0 0) 0x7f8eb00d0500 con 0x7f8eb015bd40 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.034+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb00cc870 con 0x7f8eb015bd40 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.034+0000 7f8ea7fff640 1 -- 172.21.15.6:0/2517184698 >> v1:172.21.15.92:6789/0 conn(0x7f8eb015bd40 legacy=0x7f8eb00d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.034+0000 7f8ea7fff640 1 -- 172.21.15.6:0/2517184698 >> v1:172.21.15.6:6789/0 conn(0x7f8eb00cfcd0 legacy=0x7f8eb00cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.034+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3714930200 0 0) 0x7f8e9c003520 con 0x7f8eb00cfcd0 2024-03-18T14:43:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.034+0000 7f8ea7fff640 1 -- 172.21.15.6:0/2517184698 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb00ccb10 con 0x7f8eb00d1f60 2024-03-18T14:43:27.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.035+0000 7f8ea7fff640 1 --2- 172.21.15.6:0/2517184698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00d1f60 0x7f8eb00cf790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8e980097d0 tx=0x7f8e9802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86d0eb7260e1bac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.035+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98037070 con 0x7f8eb00d1f60 2024-03-18T14:43:27.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.035+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e98030510 con 0x7f8eb00d1f60 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.035+0000 7f8eaeffd640 1 -- 172.21.15.6:0/2517184698 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80019f0 con 0x7f8eb00d1f60 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.036+0000 7f8eb609a640 1 -- 172.21.15.6:0/2517184698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00d1f60 msgr2=0x7f8eb00cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.036+0000 7f8eb609a640 1 --2- 172.21.15.6:0/2517184698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00d1f60 0x7f8eb00cf790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8e980097d0 tx=0x7f8e9802b770 comp rx=0 tx=0).stop 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.036+0000 7f8eb609a640 1 -- 172.21.15.6:0/2517184698 shutdown_connections 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.036+0000 7f8eb609a640 1 --2- 172.21.15.6:0/2517184698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00d1f60 0x7f8eb00cf790 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.036+0000 7f8eb609a640 1 -- 172.21.15.6:0/2517184698 >> 172.21.15.6:0/2517184698 conn(0x7f8eb009ae20 msgr2=0x7f8eb008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.036+0000 7f8eb609a640 1 -- 172.21.15.6:0/2517184698 shutdown_connections 2024-03-18T14:43:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.036+0000 7f8eb609a640 1 -- 172.21.15.6:0/2517184698 wait complete. 2024-03-18T14:43:27.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.037+0000 7f8eb609a640 1 Processor -- start 2024-03-18T14:43:27.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.037+0000 7f8eb609a640 1 -- start start 2024-03-18T14:43:27.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.037+0000 7f8eb609a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00cfcd0 0x7f8eb00b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.037+0000 7f8eb609a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8eb00d1f60 0x7f8eb01f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.037+0000 7f8eb609a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb015bd40 0x7f8eb014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.037+0000 7f8eb609a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb00d02e0 con 0x7f8eb015bd40 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eb609a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb00d0160 con 0x7f8eb00d1f60 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eb609a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb015c120 con 0x7f8eb00cfcd0 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb015bd40 0x7f8eb014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb015bd40 0x7f8eb014e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34810/0 (socket says 172.21.15.6:34810) 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 -- 172.21.15.6:0/1432397922 learned_addr learned my addr 172.21.15.6:0/1432397922 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8ea7fff640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8eb00d1f60 0x7f8eb01f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaf7fe640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00cfcd0 0x7f8eb00b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 -- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00cfcd0 msgr2=0x7f8eb00b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00cfcd0 0x7f8eb00b6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 -- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8eb00d1f60 msgr2=0x7f8eb01f7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8eb00d1f60 0x7f8eb01f7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 -- 172.21.15.6:0/1432397922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb01f82a0 con 0x7f8eb015bd40 2024-03-18T14:43:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eaffff640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb015bd40 0x7f8eb014e140 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8001f30 tx=0x7f8ea800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eacff9640 1 -- 172.21.15.6:0/1432397922 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea8017070 con 0x7f8eb015bd40 2024-03-18T14:43:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eacff9640 1 -- 172.21.15.6:0/1432397922 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea8012400 con 0x7f8eb015bd40 2024-03-18T14:43:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb014e680 con 0x7f8eb015bd40 2024-03-18T14:43:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eacff9640 1 -- 172.21.15.6:0/1432397922 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea8012830 con 0x7f8eb015bd40 2024-03-18T14:43:27.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.038+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb01f41c0 con 0x7f8eb015bd40 2024-03-18T14:43:27.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.039+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eb004f620 con 0x7f8eb015bd40 2024-03-18T14:43:27.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.040+0000 7f8eacff9640 1 -- 172.21.15.6:0/1432397922 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8ea801a070 con 0x7f8eb015bd40 2024-03-18T14:43:27.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.040+0000 7f8eacff9640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8e9006bbd0 0x7f8e9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.040+0000 7f8eacff9640 1 -- 172.21.15.6:0/1432397922 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f8ea808ac60 con 0x7f8eb015bd40 2024-03-18T14:43:27.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.041+0000 7f8eaf7fe640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8e9006bbd0 0x7f8e9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.041+0000 7f8eaf7fe640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8e9006bbd0 0x7f8e9006e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8ea00011e0 tx=0x7f8ea0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.045+0000 7f8eacff9640 1 -- 172.21.15.6:0/1432397922 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8eb014e680 con 0x7f8eb015bd40 2024-03-18T14:43:27.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.228+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8eb008b1f0 con 0x7f8eb015bd40 2024-03-18T14:43:27.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.229+0000 7f8eacff9640 1 -- 172.21.15.6:0/1432397922 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+18707 (secure 0 0 0) 0x7f8ea805de20 con 0x7f8eb015bd40 2024-03-18T14:43:27.230 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:27.230 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:17.388447+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8e9006bbd0 msgr2=0x7f8e9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8e9006bbd0 0x7f8e9006e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8ea00011e0 tx=0x7f8ea0009040 comp rx=0 tx=0).stop 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb015bd40 msgr2=0x7f8eb014e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb015bd40 0x7f8eb014e140 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8001f30 tx=0x7f8ea800a430 comp rx=0 tx=0).stop 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 shutdown_connections 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8e9006bbd0 0x7f8e9006e090 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8eb015bd40 0x7f8eb014e140 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8eb00d1f60 0x7f8eb01f7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 --2- 172.21.15.6:0/1432397922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8eb00cfcd0 0x7f8eb00b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 >> 172.21.15.6:0/1432397922 conn(0x7f8eb009ae20 msgr2=0x7f8eb015a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 shutdown_connections 2024-03-18T14:43:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.231+0000 7f8eb609a640 1 -- 172.21.15.6:0/1432397922 wait complete. 2024-03-18T14:43:27.244 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T14:43:27.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:27.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.344+0000 7f7f9aa7e640 1 Processor -- start 2024-03-18T14:43:27.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.344+0000 7f7f9aa7e640 1 -- start start 2024-03-18T14:43:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.345+0000 7f7f9aa7e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7f94162380 0x7f7f940d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.345+0000 7f7f9aa7e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f940b9660 con 0x7f7f9415e020 2024-03-18T14:43:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.345+0000 7f7f9aa7e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f94162810 con 0x7f7f940ced00 2024-03-18T14:43:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.345+0000 7f7f9aa7e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7f940d2270 con 0x7f7f94162380 2024-03-18T14:43:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.345+0000 7f7f9881a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7f9415e020 0x7f7f940d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59828/0 (socket says 172.21.15.6:59828) 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.345+0000 7f7f9881a640 1 -- 172.21.15.6:0/1334377371 learned_addr learned my addr 172.21.15.6:0/1334377371 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9901b640 1 --2- 172.21.15.6:0/1334377371 >> v2:172.21.15.92:3300/0 conn(0x7f7f94162380 0x7f7f940d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195430486 0 0) 0x7f7f940b9660 con 0x7f7f9415e020 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f7c003520 con 0x7f7f9415e020 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 283067811 0 0) 0x7f7f7c003520 con 0x7f7f9415e020 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 >> v2:172.21.15.92:3300/0 conn(0x7f7f94162380 msgr2=0x7f7f940d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 --2- 172.21.15.6:0/1334377371 >> v2:172.21.15.92:3300/0 conn(0x7f7f94162380 0x7f7f940d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 >> v1:172.21.15.92:6789/0 conn(0x7f7f940ced00 legacy=0x7f7f940d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f940ce390 con 0x7f7f9415e020 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f7f8c001a10 con 0x7f7f9415e020 2024-03-18T14:43:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 >> v1:172.21.15.6:6789/0 conn(0x7f7f9415e020 legacy=0x7f7f940d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f7c0043b0 0x7f7f7c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f7c005120 0x7f7f7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7f940d2270 con 0x7f7f7c0039b0 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c00c060 con 0x7f7f7c0043b0 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c00c260 con 0x7f7f7c005120 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9881a640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f93fff640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f7c0043b0 0x7f7f7c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9901b640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f7c005120 0x7f7f7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9881a640 1 -- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f7c005120 msgr2=0x7f7f7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9881a640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f7c005120 0x7f7f7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9881a640 1 -- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f7c0043b0 msgr2=0x7f7f7c004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:27.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9881a640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f7c0043b0 0x7f7f7c004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9881a640 1 -- 172.21.15.6:0/1334377371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c003520 con 0x7f7f7c0039b0 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9901b640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f7c005120 0x7f7f7c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f93fff640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f7c0043b0 0x7f7f7c004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.346+0000 7f7f9901b640 1 -- 172.21.15.6:0/1334377371 reap_dead start 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.347+0000 7f7f9881a640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f7c003d90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00a880 tx=0x7f7f7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7391255cb0915d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.347+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00a4d0 con 0x7f7f7c0039b0 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.347+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f8c0072d0 con 0x7f7f7c0039b0 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.347+0000 7f7f937fe640 1 -- 172.21.15.6:0/1334377371 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c02ce60 con 0x7f7f7c0039b0 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.348+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/1334377371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 msgr2=0x7f7f7c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.348+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f7c003d90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00a880 tx=0x7f7f7c005830 comp rx=0 tx=0).stop 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.348+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/1334377371 shutdown_connections 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.348+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/1334377371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f7c003d90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.348+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/1334377371 >> 172.21.15.6:0/1334377371 conn(0x7f7f94072ce0 msgr2=0x7f7f940730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.348+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/1334377371 shutdown_connections 2024-03-18T14:43:27.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.348+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/1334377371 wait complete. 2024-03-18T14:43:27.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.349+0000 7f7f9aa7e640 1 Processor -- start 2024-03-18T14:43:27.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.349+0000 7f7f9aa7e640 1 -- start start 2024-03-18T14:43:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9aa7e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f940b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9aa7e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f9415b6a0 0x7f7f9414e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9aa7e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f9415b150 0x7f7f9415fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9aa7e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7f940ce890 con 0x7f7f7c0039b0 2024-03-18T14:43:27.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9aa7e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7f940b9be0 con 0x7f7f9415b6a0 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9aa7e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7f940cf2c0 con 0x7f7f9415b150 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f940b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f940b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34832/0 (socket says 172.21.15.6:34832) 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 -- 172.21.15.6:0/3937505344 learned_addr learned my addr 172.21.15.6:0/3937505344 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f93fff640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f9415b6a0 0x7f7f9414e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9901b640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f9415b150 0x7f7f9415fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 -- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f9415b150 msgr2=0x7f7f9415fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f9415b150 0x7f7f9415fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 -- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f9415b6a0 msgr2=0x7f7f9414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f9415b6a0 0x7f7f9414e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 -- 172.21.15.6:0/3937505344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f941612b0 con 0x7f7f7c0039b0 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9901b640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f9415b150 0x7f7f9415fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:27.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.350+0000 7f7f9881a640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f940b5900 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00a970 tx=0x7f7f8c001de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.351+0000 7f7f917fa640 1 -- 172.21.15.6:0/3937505344 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c039040 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.351+0000 7f7f917fa640 1 -- 172.21.15.6:0/3937505344 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f8c043070 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.351+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9414e600 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.351+0000 7f7f917fa640 1 -- 172.21.15.6:0/3937505344 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00a4d0 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.351+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f940b4b40 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.352+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f9408afe0 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.352+0000 7f7f917fa640 1 -- 172.21.15.6:0/3937505344 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7f8c004b60 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.353+0000 7f7f917fa640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f8006bbd0 0x7f7f8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.353+0000 7f7f917fa640 1 -- 172.21.15.6:0/3937505344 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f7f8c0ac5e0 con 0x7f7f7c0039b0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.353+0000 7f7f93fff640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f8006bbd0 0x7f7f8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.353+0000 7f7f93fff640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f8006bbd0 0x7f7f8006e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7f940d2930 tx=0x7f7f88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.357+0000 7f7f917fa640 1 -- 172.21.15.6:0/3937505344 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7f8c04c020 con 0x7f7f7c0039b0 2024-03-18T14:43:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.578+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f9404f620 con 0x7f7f7c0039b0 2024-03-18T14:43:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.578+0000 7f7f917fa640 1 -- 172.21.15.6:0/3937505344 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5171 (secure 0 0 0) 0x7f7f8c07ae30 con 0x7f7f7c0039b0 2024-03-18T14:43:27.580 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:27.580 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:42:56.672813+0000","modified":"2024-03-18T14:43:04.715542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5529":{"gid":5529,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2677248951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2677248951},{"type":"v1","addr":"172.21.15.6:6834","nonce":2677248951}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5529,"qdb_cluster":[5529]},"id":2}]} 2024-03-18T14:43:27.580 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-18T14:43:27.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f8006bbd0 msgr2=0x7f7f8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f8006bbd0 0x7f7f8006e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7f940d2930 tx=0x7f7f88009040 comp rx=0 tx=0).stop 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 msgr2=0x7f7f940b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f940b5900 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00a970 tx=0x7f7f8c001de0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 shutdown_connections 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7f8006bbd0 0x7f7f8006e090 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f9415b150 0x7f7f9415fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f9415b6a0 0x7f7f9414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 --2- 172.21.15.6:0/3937505344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f7c0039b0 0x7f7f940b5900 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 >> 172.21.15.6:0/3937505344 conn(0x7f7f94072ce0 msgr2=0x7f7f9406c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 shutdown_connections 2024-03-18T14:43:27.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.581+0000 7f7f9aa7e640 1 -- 172.21.15.6:0/3937505344 wait complete. 2024-03-18T14:43:27.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:27.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.693+0000 7fdc21710640 1 Processor -- start 2024-03-18T14:43:27.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.694+0000 7fdc21710640 1 -- start start 2024-03-18T14:43:27.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.694+0000 7fdc21710640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc1c15e170 0x7fdc1c0b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.694+0000 7fdc21710640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc1c0d3330 0x7fdc1c0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.694+0000 7fdc21710640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc1c0cef50 con 0x7fdc1c0d37c0 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.694+0000 7fdc21710640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fdc1c0b9b90 con 0x7fdc1c0d3330 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.694+0000 7fdc21710640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdc1c0cf230 con 0x7fdc1c15e170 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.694+0000 7fdc1a7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc1c0d3330 0x7fdc1c0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1a7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc1c0d3330 0x7fdc1c0d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34846/0 (socket says 172.21.15.6:34846) 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdc1c0d37c0 0x7fdc1c0b6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59840/0 (socket says 172.21.15.6:59840) 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1a7fc640 1 -- 172.21.15.6:0/1153233133 learned_addr learned my addr 172.21.15.6:0/1153233133 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1affd640 1 --2- 172.21.15.6:0/1153233133 >> v2:172.21.15.92:3300/0 conn(0x7fdc1c15e170 0x7fdc1c0b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1a7fc640 1 -- 172.21.15.6:0/1153233133 >> v2:172.21.15.92:3300/0 conn(0x7fdc1c15e170 msgr2=0x7fdc1c0b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1a7fc640 1 --2- 172.21.15.6:0/1153233133 >> v2:172.21.15.92:3300/0 conn(0x7fdc1c15e170 0x7fdc1c0b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1a7fc640 1 -- 172.21.15.6:0/1153233133 >> v1:172.21.15.6:6789/0 conn(0x7fdc1c0d37c0 legacy=0x7fdc1c0b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc19ffb640 1 -- 172.21.15.6:0/1153233133 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856406005 0 0) 0x7fdc1c0cef50 con 0x7fdc1c0d37c0 2024-03-18T14:43:27.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.695+0000 7fdc1a7fc640 1 -- 172.21.15.6:0/1153233133 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c0d2b30 con 0x7fdc1c0d3330 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.696+0000 7fdc1a7fc640 1 --2- 172.21.15.6:0/1153233133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0d0680 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c08c3a0 tx=0x7fdc1002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc47ab4b1980fd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.696+0000 7fdc19ffb640 1 -- 172.21.15.6:0/1153233133 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10034070 con 0x7fdc1c0d3330 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.696+0000 7fdc19ffb640 1 -- 172.21.15.6:0/1153233133 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc10030860 con 0x7fdc1c0d3330 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.696+0000 7fdc19ffb640 1 -- 172.21.15.6:0/1153233133 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10030c30 con 0x7fdc1c0d3330 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.696+0000 7fdc21710640 1 -- 172.21.15.6:0/1153233133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 msgr2=0x7fdc1c0d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.696+0000 7fdc21710640 1 --2- 172.21.15.6:0/1153233133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0d0680 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c08c3a0 tx=0x7fdc1002b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.697+0000 7fdc21710640 1 -- 172.21.15.6:0/1153233133 shutdown_connections 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.697+0000 7fdc21710640 1 --2- 172.21.15.6:0/1153233133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0d0680 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.697+0000 7fdc21710640 1 --2- 172.21.15.6:0/1153233133 >> v2:172.21.15.92:3300/0 conn(0x7fdc1c15e170 0x7fdc1c0b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.697+0000 7fdc21710640 1 -- 172.21.15.6:0/1153233133 >> 172.21.15.6:0/1153233133 conn(0x7fdc1c06c900 msgr2=0x7fdc1c06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.697+0000 7fdc21710640 1 -- 172.21.15.6:0/1153233133 shutdown_connections 2024-03-18T14:43:27.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.697+0000 7fdc21710640 1 -- 172.21.15.6:0/1153233133 wait complete. 2024-03-18T14:43:27.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.698+0000 7fdc21710640 1 Processor -- start 2024-03-18T14:43:27.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.698+0000 7fdc21710640 1 -- start start 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc21710640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc21710640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1c0d37c0 0x7fdc1c0d3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc21710640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc1c15e170 0x7fdc1c0d4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc21710640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c0ceb70 con 0x7fdc1c0d3330 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc21710640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c0d2820 con 0x7fdc1c0d37c0 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc21710640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c0cecf0 con 0x7fdc1c15e170 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34862/0 (socket says 172.21.15.6:34862) 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc1affd640 1 -- 172.21.15.6:0/1010174998 learned_addr learned my addr 172.21.15.6:0/1010174998 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc1a7fc640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1c0d37c0 0x7fdc1c0d3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.699+0000 7fdc1b7fe640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc1c15e170 0x7fdc1c0d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc1affd640 1 -- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc1c15e170 msgr2=0x7fdc1c0d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc1affd640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc1c15e170 0x7fdc1c0d4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc1affd640 1 -- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1c0d37c0 msgr2=0x7fdc1c0d3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc1affd640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1c0d37c0 0x7fdc1c0d3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc1affd640 1 -- 172.21.15.6:0/1010174998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c161590 con 0x7fdc1c0d3330 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc1affd640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0c6f10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400d5f0 tx=0x7fdc1400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc037fe640 1 -- 172.21.15.6:0/1010174998 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc140125f0 con 0x7fdc1c0d3330 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc037fe640 1 -- 172.21.15.6:0/1010174998 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc14019070 con 0x7fdc1c0d3330 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c1f44e0 con 0x7fdc1c0d3330 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc037fe640 1 -- 172.21.15.6:0/1010174998 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc140129d0 con 0x7fdc1c0d3330 2024-03-18T14:43:27.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.700+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c148ed0 con 0x7fdc1c0d3330 2024-03-18T14:43:27.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.701+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1c08afe0 con 0x7fdc1c0d3330 2024-03-18T14:43:27.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.702+0000 7fdc037fe640 1 -- 172.21.15.6:0/1010174998 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdc14012b70 con 0x7fdc1c0d3330 2024-03-18T14:43:27.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.702+0000 7fdc037fe640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdc0806bc00 0x7fdc0806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:27.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.702+0000 7fdc037fe640 1 -- 172.21.15.6:0/1010174998 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7fdc1408b0e0 con 0x7fdc1c0d3330 2024-03-18T14:43:27.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.702+0000 7fdc1a7fc640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdc0806bc00 0x7fdc0806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:27.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.703+0000 7fdc1a7fc640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdc0806bc00 0x7fdc0806e0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdc10036000 tx=0x7fdc1003b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:27.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.706+0000 7fdc037fe640 1 -- 172.21.15.6:0/1010174998 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdc14059930 con 0x7fdc1c0d3330 2024-03-18T14:43:27.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.926+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc1c04f620 con 0x7fdc1c0d3330 2024-03-18T14:43:27.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.927+0000 7fdc037fe640 1 -- 172.21.15.6:0/1010174998 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5171 (secure 0 0 0) 0x7fdc14059230 con 0x7fdc1c0d3330 2024-03-18T14:43:27.928 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:27.928 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:42:56.672813+0000","modified":"2024-03-18T14:43:04.715542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5529},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5529":{"gid":5529,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2677248951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2677248951},{"type":"v1","addr":"172.21.15.6:6834","nonce":2677248951}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5529,"qdb_cluster":[5529]},"id":2}]} 2024-03-18T14:43:27.928 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 16 2024-03-18T14:43:27.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdc0806bc00 msgr2=0x7fdc0806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdc0806bc00 0x7fdc0806e0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdc10036000 tx=0x7fdc1003b040 comp rx=0 tx=0).stop 2024-03-18T14:43:27.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 msgr2=0x7fdc1c0c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:27.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0c6f10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400d5f0 tx=0x7fdc1400dab0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 shutdown_connections 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdc0806bc00 0x7fdc0806e0c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc1c15e170 0x7fdc1c0d4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1c0d37c0 0x7fdc1c0d3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.928+0000 7fdc21710640 1 --2- 172.21.15.6:0/1010174998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c0d3330 0x7fdc1c0c6f10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.929+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 >> 172.21.15.6:0/1010174998 conn(0x7fdc1c06c900 msgr2=0x7fdc1c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.929+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 shutdown_connections 2024-03-18T14:43:27.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:27.929+0000 7fdc21710640 1 -- 172.21.15.6:0/1010174998 wait complete. 2024-03-18T14:43:27.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:28.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.048+0000 7f90965c4640 1 Processor -- start 2024-03-18T14:43:28.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.048+0000 7f90965c4640 1 -- start start 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f90965c4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f909015bd40 0x7f90900d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f90965c4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90900d1d90 0x7f90900cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f90965c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90900cfcd0 0x7f90900ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f90965c4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f90900cc870 con 0x7f90900d1d90 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f90965c4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f90900d0500 con 0x7f909015bd40 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f90965c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f90900d07e0 con 0x7f90900cfcd0 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f908f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90900d1d90 0x7f90900cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f908f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90900d1d90 0x7f90900cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34868/0 (socket says 172.21.15.6:34868) 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.049+0000 7f908f7fe640 1 -- 172.21.15.6:0/2740747222 learned_addr learned my addr 172.21.15.6:0/2740747222 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f9094b61640 1 --2- 172.21.15.6:0/2740747222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90900cfcd0 0x7f90900ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f908ffff640 1 --2- 172.21.15.6:0/2740747222 >> v2:172.21.15.92:3300/0 conn(0x7f909015bd40 0x7f90900d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f908f7fe640 1 -- 172.21.15.6:0/2740747222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90900cfcd0 msgr2=0x7f90900ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f908f7fe640 1 --2- 172.21.15.6:0/2740747222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90900cfcd0 0x7f90900ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f908f7fe640 1 -- 172.21.15.6:0/2740747222 >> v2:172.21.15.92:3300/0 conn(0x7f909015bd40 msgr2=0x7f90900d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f908f7fe640 1 --2- 172.21.15.6:0/2740747222 >> v2:172.21.15.92:3300/0 conn(0x7f909015bd40 0x7f90900d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f908f7fe640 1 -- 172.21.15.6:0/2740747222 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90900ccb10 con 0x7f90900d1d90 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f9094b61640 1 --2- 172.21.15.6:0/2740747222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90900cfcd0 0x7f90900ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.050+0000 7f908f7fe640 1 --2- 172.21.15.6:0/2740747222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900d1d90 0x7f90900cf790 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f909008bae0 tx=0x7f908402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a8f1f3a0a84307 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f908effd640 1 -- 172.21.15.6:0/2740747222 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084035020 con 0x7f90900d1d90 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f908effd640 1 -- 172.21.15.6:0/2740747222 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9084031870 con 0x7f90900d1d90 2024-03-18T14:43:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f908effd640 1 -- 172.21.15.6:0/2740747222 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084031c40 con 0x7f90900d1d90 2024-03-18T14:43:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 -- 172.21.15.6:0/2740747222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900d1d90 msgr2=0x7f90900cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 --2- 172.21.15.6:0/2740747222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900d1d90 0x7f90900cf790 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f909008bae0 tx=0x7f908402b910 comp rx=0 tx=0).stop 2024-03-18T14:43:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 -- 172.21.15.6:0/2740747222 shutdown_connections 2024-03-18T14:43:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 --2- 172.21.15.6:0/2740747222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90900cfcd0 0x7f90900ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 --2- 172.21.15.6:0/2740747222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900d1d90 0x7f90900cf790 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 --2- 172.21.15.6:0/2740747222 >> v2:172.21.15.92:3300/0 conn(0x7f909015bd40 0x7f90900d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 -- 172.21.15.6:0/2740747222 >> 172.21.15.6:0/2740747222 conn(0x7f909009ae20 msgr2=0x7f909008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 -- 172.21.15.6:0/2740747222 shutdown_connections 2024-03-18T14:43:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.051+0000 7f90965c4640 1 -- 172.21.15.6:0/2740747222 wait complete. 2024-03-18T14:43:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.053+0000 7f90965c4640 1 Processor -- start 2024-03-18T14:43:28.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.053+0000 7f90965c4640 1 -- start start 2024-03-18T14:43:28.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.054+0000 7f90965c4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900cfcd0 0x7f90900afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.054+0000 7f90965c4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90900d1d90 0x7f90900cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.054+0000 7f90965c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909015bd40 0x7f9090070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.054+0000 7f90965c4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f90900d0c40 con 0x7f90900cfcd0 2024-03-18T14:43:28.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.054+0000 7f90965c4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f90900d0ac0 con 0x7f90900d1d90 2024-03-18T14:43:28.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.054+0000 7f90965c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f90900d0140 con 0x7f909015bd40 2024-03-18T14:43:28.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900cfcd0 0x7f90900afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900cfcd0 0x7f90900afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34872/0 (socket says 172.21.15.6:34872) 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 -- 172.21.15.6:0/2578742028 learned_addr learned my addr 172.21.15.6:0/2578742028 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908f7fe640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90900d1d90 0x7f90900cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f9094b61640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909015bd40 0x7f9090070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 -- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909015bd40 msgr2=0x7f9090070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909015bd40 0x7f9090070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 -- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90900d1d90 msgr2=0x7f90900cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90900d1d90 0x7f90900cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 -- 172.21.15.6:0/2578742028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90900af2a0 con 0x7f90900cfcd0 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908f7fe640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90900d1d90 0x7f90900cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908ffff640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900cfcd0 0x7f90900afe40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f908000d800 tx=0x7f908000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:28.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908cff9640 1 -- 172.21.15.6:0/2578742028 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908000b6c0 con 0x7f90900cfcd0 2024-03-18T14:43:28.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908cff9640 1 -- 172.21.15.6:0/2578742028 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9080019070 con 0x7f90900cfcd0 2024-03-18T14:43:28.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90900af580 con 0x7f90900cfcd0 2024-03-18T14:43:28.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f908cff9640 1 -- 172.21.15.6:0/2578742028 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908000ba80 con 0x7f90900cfcd0 2024-03-18T14:43:28.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.055+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90900b5be0 con 0x7f90900cfcd0 2024-03-18T14:43:28.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.057+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f909009fa90 con 0x7f90900cfcd0 2024-03-18T14:43:28.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.057+0000 7f908cff9640 1 -- 172.21.15.6:0/2578742028 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f908000bc20 con 0x7f90900cfcd0 2024-03-18T14:43:28.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.057+0000 7f908cff9640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f907c06bbd0 0x7f907c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.057+0000 7f908cff9640 1 -- 172.21.15.6:0/2578742028 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f908008b180 con 0x7f90900cfcd0 2024-03-18T14:43:28.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.057+0000 7f908f7fe640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f907c06bbd0 0x7f907c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.058+0000 7f908f7fe640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f907c06bbd0 0x7f907c06e090 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9084036000 tx=0x7f908403c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:28.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.061+0000 7f908cff9640 1 -- 172.21.15.6:0/2578742028 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f90800599d0 con 0x7f90900cfcd0 2024-03-18T14:43:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.335+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90900b2840 con 0x7f90900cfcd0 2024-03-18T14:43:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.336+0000 7f908cff9640 1 -- 172.21.15.6:0/2578742028 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+18707 (secure 0 0 0) 0x7f90800592d0 con 0x7f90900cfcd0 2024-03-18T14:43:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:28.338 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:17.388447+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:42:54.939648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T14:42:55.638120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-18T14:43:03.985735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f907c06bbd0 msgr2=0x7f907c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f907c06bbd0 0x7f907c06e090 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9084036000 tx=0x7f908403c040 comp rx=0 tx=0).stop 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900cfcd0 msgr2=0x7f90900afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900cfcd0 0x7f90900afe40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f908000d800 tx=0x7f908000dcc0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 shutdown_connections 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f907c06bbd0 0x7f907c06e090 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909015bd40 0x7f9090070eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90900d1d90 0x7f90900cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 --2- 172.21.15.6:0/2578742028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90900cfcd0 0x7f90900afe40 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 >> 172.21.15.6:0/2578742028 conn(0x7f909009ae20 msgr2=0x7f90900b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 shutdown_connections 2024-03-18T14:43:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.338+0000 7f90965c4640 1 -- 172.21.15.6:0/2578742028 wait complete. 2024-03-18T14:43:28.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T14:43:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.491+0000 7f98f2e0b640 1 Processor -- start 2024-03-18T14:43:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.491+0000 7f98f2e0b640 1 -- start start 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f2e0b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f98ec15ad20 0x7f98ec15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f2e0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec0ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f2e0b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98ec0b2870 con 0x7f98ec06ead0 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f2e0b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f98ec15b710 con 0x7f98ec15ad20 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f2e0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f98ec15b9f0 con 0x7f98ec0cdce0 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f13a8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f98ec06ead0 0x7f98ec0b2d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59856/0 (socket says 172.21.15.6:59856) 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f13a8640 1 -- 172.21.15.6:0/1112511989 learned_addr learned my addr 172.21.15.6:0/1112511989 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f0ba7640 1 --2- 172.21.15.6:0/1112511989 >> v2:172.21.15.92:3300/0 conn(0x7f98ec15ad20 0x7f98ec15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98ebfff640 1 --2- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec0ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98eb7fe640 1 -- 172.21.15.6:0/1112511989 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1139591197 0 0) 0x7f98ec0b2870 con 0x7f98ec06ead0 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98eb7fe640 1 -- 172.21.15.6:0/1112511989 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98d4003520 con 0x7f98ec06ead0 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f0ba7640 1 -- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 msgr2=0x7f98ec0ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f0ba7640 1 --2- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec0ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f0ba7640 1 -- 172.21.15.6:0/1112511989 >> v1:172.21.15.6:6789/0 conn(0x7f98ec06ead0 legacy=0x7f98ec0b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98eb7fe640 1 -- 172.21.15.6:0/1112511989 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1818530861 0 0) 0x7f98d4003520 con 0x7f98ec06ead0 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98f0ba7640 1 -- 172.21.15.6:0/1112511989 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ec15bcd0 con 0x7f98ec15ad20 2024-03-18T14:43:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.493+0000 7f98ebfff640 1 --2- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec0ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.494+0000 7f98f0ba7640 1 --2- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec15b100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009d90 tx=0x7f98e402bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f17443bc4a6cbaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.494+0000 7f98eb7fe640 1 -- 172.21.15.6:0/1112511989 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e4039070 con 0x7f98ec15ad20 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.494+0000 7f98eb7fe640 1 -- 172.21.15.6:0/1112511989 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e402c810 con 0x7f98ec15ad20 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98eb7fe640 1 -- 172.21.15.6:0/1112511989 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc001670 con 0x7f98ec15ad20 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 msgr2=0x7f98ec15b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec15b100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009d90 tx=0x7f98e402bce0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1112511989 shutdown_connections 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec0ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1112511989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec15b100 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1112511989 >> 172.21.15.6:0/1112511989 conn(0x7f98ec0b1570 msgr2=0x7f98ec0b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:28.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1112511989 shutdown_connections 2024-03-18T14:43:28.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.495+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1112511989 wait complete. 2024-03-18T14:43:28.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.496+0000 7f98f2e0b640 1 Processor -- start 2024-03-18T14:43:28.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.496+0000 7f98f2e0b640 1 -- start start 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f2e0b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98ec06ead0 0x7f98ec14cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f2e0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec141f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f2e0b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f2e0b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec0ce2c0 con 0x7f98ec06ead0 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f2e0b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec0ce140 con 0x7f98ec15ad20 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f2e0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f98ec0aef10 con 0x7f98ec0cdce0 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f0ba7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98ec06ead0 0x7f98ec14cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f0ba7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98ec06ead0 0x7f98ec14cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34888/0 (socket says 172.21.15.6:34888) 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.497+0000 7f98f0ba7640 1 -- 172.21.15.6:0/1373906832 learned_addr learned my addr 172.21.15.6:0/1373906832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f13a8640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98ebfff640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec141f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f0ba7640 1 -- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 msgr2=0x7f98ec141f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f0ba7640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec141f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f0ba7640 1 -- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 msgr2=0x7f98ec142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f0ba7640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec142720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f0ba7640 1 -- 172.21.15.6:0/1373906832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ec14d300 con 0x7f98ec06ead0 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f13a8640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f0ba7640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98ec06ead0 0x7f98ec14cdc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009eb0 tx=0x7f98e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98e97fa640 1 -- 172.21.15.6:0/1373906832 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e4039050 con 0x7f98ec06ead0 2024-03-18T14:43:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98e97fa640 1 -- 172.21.15.6:0/1373906832 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e4045070 con 0x7f98ec06ead0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98e97fa640 1 -- 172.21.15.6:0/1373906832 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e4004670 con 0x7f98ec06ead0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ec0cee50 con 0x7f98ec06ead0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.498+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ec15a140 con 0x7f98ec06ead0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.499+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ec04f5b0 con 0x7f98ec06ead0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.500+0000 7f98e97fa640 1 -- 172.21.15.6:0/1373906832 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f98e404a460 con 0x7f98ec06ead0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.500+0000 7f98e97fa640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98d806bbd0 0x7f98d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.500+0000 7f98e97fa640 1 -- 172.21.15.6:0/1373906832 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6525+0+0 (secure 0 0 0) 0x7f98e40ad5d0 con 0x7f98ec06ead0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.500+0000 7f98ebfff640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98d806bbd0 0x7f98d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.501+0000 7f98ebfff640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98d806bbd0 0x7f98d806e090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f98ec0ae790 tx=0x7f98e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.504+0000 7f98e97fa640 1 -- 172.21.15.6:0/1373906832 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f98e403d360 con 0x7f98ec06ead0 2024-03-18T14:43:28.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.897+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f98ec151fe0 con 0x7f98ec06ead0 2024-03-18T14:43:28.899 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-18T14:43:28.898+0000 7f3765211640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T14:43:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.899+0000 7f98e97fa640 1 -- 172.21.15.6:0/1373906832 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f98e407beb0 con 0x7f98ec06ead0 2024-03-18T14:43:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T14:43:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98d806bbd0 msgr2=0x7f98d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98d806bbd0 0x7f98d806e090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f98ec0ae790 tx=0x7f98e0008040 comp rx=0 tx=0).stop 2024-03-18T14:43:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98ec06ead0 msgr2=0x7f98ec14cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98ec06ead0 0x7f98ec14cdc0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009eb0 tx=0x7f98e4004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 shutdown_connections 2024-03-18T14:43:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98d806bbd0 0x7f98d806e090 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98ec15ad20 0x7f98ec142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98ec0cdce0 0x7f98ec141f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 --2- 172.21.15.6:0/1373906832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98ec06ead0 0x7f98ec14cdc0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 >> 172.21.15.6:0/1373906832 conn(0x7f98ec0b1570 msgr2=0x7f98ec072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 shutdown_connections 2024-03-18T14:43:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:28.902+0000 7f98f2e0b640 1 -- 172.21.15.6:0/1373906832 wait complete. 2024-03-18T14:43:28.905 INFO:tasks.ceph.mds.c.smithi006.stderr: -3370> 2024-03-18T14:43:01.988+0000 7f2cb0137640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T14:43:28.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T14:43:28.935 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-18T14:43:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.019+0000 7ff10820b640 1 Processor -- start 2024-03-18T14:43:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.019+0000 7ff10820b640 1 -- start start 2024-03-18T14:43:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff10820b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff10015dfd0 0x7ff1000b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff10820b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b7e00 0x7ff1000b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff10820b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10015e3b0 con 0x7ff1000b5740 2024-03-18T14:43:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff10820b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff1000b8470 con 0x7ff10015dfd0 2024-03-18T14:43:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff10820b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff1000b8750 con 0x7ff1000b7e00 2024-03-18T14:43:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff105fa7640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff10015dfd0 0x7ff1000b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff1067a8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b7e00 0x7ff1000b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff1067a8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b7e00 0x7ff1000b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54050/0 (socket says 172.21.15.6:54050) 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff1067a8640 1 -- 172.21.15.6:0/2211280636 learned_addr learned my addr 172.21.15.6:0/2211280636 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.020+0000 7ff1057a6640 1 --1- 172.21.15.6:0/2211280636 >> v1:172.21.15.92:6789/0 conn(0x7ff1000b5740 0x7ff1000d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45872/0 (socket says 172.21.15.6:45872) 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.021+0000 7ff1067a8640 1 -- 172.21.15.6:0/2211280636 >> v2:172.21.15.92:3300/0 conn(0x7ff10015dfd0 msgr2=0x7ff1000b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.021+0000 7ff1067a8640 1 --2- 172.21.15.6:0/2211280636 >> v2:172.21.15.92:3300/0 conn(0x7ff10015dfd0 0x7ff1000b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.021+0000 7ff1067a8640 1 -- 172.21.15.6:0/2211280636 >> v1:172.21.15.92:6789/0 conn(0x7ff1000b5740 legacy=0x7ff1000d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.021+0000 7ff1067a8640 1 -- 172.21.15.6:0/2211280636 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1000b8a30 con 0x7ff1000b7e00 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff1067a8640 1 --2- 172.21.15.6:0/2211280636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b7e00 0x7ff1000b5bd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009e50 tx=0x7ff0f802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9d6c51c7ef651bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff104fa5640 1 -- 172.21.15.6:0/2211280636 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f802cb00 con 0x7ff1000b7e00 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff104fa5640 1 -- 172.21.15.6:0/2211280636 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f802cca0 con 0x7ff1000b7e00 2024-03-18T14:43:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff104fa5640 1 -- 172.21.15.6:0/2211280636 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8034610 con 0x7ff1000b7e00 2024-03-18T14:43:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 -- 172.21.15.6:0/2211280636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b7e00 msgr2=0x7ff1000b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 --2- 172.21.15.6:0/2211280636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b7e00 0x7ff1000b5bd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009e50 tx=0x7ff0f802bf60 comp rx=0 tx=0).stop 2024-03-18T14:43:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 -- 172.21.15.6:0/2211280636 shutdown_connections 2024-03-18T14:43:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 --2- 172.21.15.6:0/2211280636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b7e00 0x7ff1000b5bd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 --2- 172.21.15.6:0/2211280636 >> v2:172.21.15.92:3300/0 conn(0x7ff10015dfd0 0x7ff1000b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 -- 172.21.15.6:0/2211280636 >> 172.21.15.6:0/2211280636 conn(0x7ff100072d10 msgr2=0x7ff100073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 -- 172.21.15.6:0/2211280636 shutdown_connections 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.022+0000 7ff10820b640 1 -- 172.21.15.6:0/2211280636 wait complete. 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.023+0000 7ff10820b640 1 Processor -- start 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff10820b640 1 -- start start 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff10820b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b5740 0x7ff1000b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff10820b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1000b7e00 0x7ff100160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff10820b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10015dfd0 0x7ff10015b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff10820b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff1000d2550 con 0x7ff10015dfd0 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff10820b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff1000b9b90 con 0x7ff1000b7e00 2024-03-18T14:43:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff10820b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff1000d26d0 con 0x7ff1000b5740 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff1067a8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10015dfd0 0x7ff10015b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff1067a8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10015dfd0 0x7ff10015b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34918/0 (socket says 172.21.15.6:34918) 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff1067a8640 1 -- 172.21.15.6:0/3702916665 learned_addr learned my addr 172.21.15.6:0/3702916665 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff1057a6640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1000b7e00 0x7ff100160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.024+0000 7ff105fa7640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b5740 0x7ff1000b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff1067a8640 1 -- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b5740 msgr2=0x7ff1000b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff1067a8640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b5740 0x7ff1000b19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff1067a8640 1 -- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1000b7e00 msgr2=0x7ff100160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff1067a8640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1000b7e00 0x7ff100160480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff1067a8640 1 -- 172.21.15.6:0/3702916665 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10015ea30 con 0x7ff10015dfd0 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff1067a8640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10015dfd0 0x7ff10015b0f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8036040 tx=0x7ff0f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff0f67fc640 1 -- 172.21.15.6:0/3702916665 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8043020 con 0x7ff10015dfd0 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff0f67fc640 1 -- 172.21.15.6:0/3702916665 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f802ce50 con 0x7ff10015dfd0 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10015ed10 con 0x7ff10015dfd0 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff0f67fc640 1 -- 172.21.15.6:0/3702916665 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8009510 con 0x7ff10015dfd0 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.025+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10015f2d0 con 0x7ff10015dfd0 2024-03-18T14:43:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.026+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1000751b0 con 0x7ff10015dfd0 2024-03-18T14:43:29.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.026+0000 7ff0f67fc640 1 -- 172.21.15.6:0/3702916665 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff0f8046020 con 0x7ff10015dfd0 2024-03-18T14:43:29.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.027+0000 7ff0f67fc640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0e406bbd0 0x7ff0e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:29.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.027+0000 7ff0f67fc640 1 -- 172.21.15.6:0/3702916665 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6611+0+0 (secure 0 0 0) 0x7ff0f80ad230 con 0x7ff10015dfd0 2024-03-18T14:43:29.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.027+0000 7ff105fa7640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0e406bbd0 0x7ff0e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:29.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.027+0000 7ff105fa7640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0e406bbd0 0x7ff0e406e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1001f6d80 tx=0x7ff0f0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:29.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.031+0000 7ff0f67fc640 1 -- 172.21.15.6:0/3702916665 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff0f804d030 con 0x7ff10015dfd0 2024-03-18T14:43:29.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.253+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff10006b910 con 0x7ff10015dfd0 2024-03-18T14:43:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.903+0000 7ff0f67fc640 1 -- 172.21.15.6:0/3702916665 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7ff0f807b340 con 0x7ff10015dfd0 2024-03-18T14:43:29.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0e406bbd0 msgr2=0x7ff0e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0e406bbd0 0x7ff0e406e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1001f6d80 tx=0x7ff0f0009190 comp rx=0 tx=0).stop 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10015dfd0 msgr2=0x7ff10015b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10015dfd0 0x7ff10015b0f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8036040 tx=0x7ff0f8004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 shutdown_connections 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff0e406bbd0 0x7ff0e406e090 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10015dfd0 0x7ff10015b0f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1000b7e00 0x7ff100160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 --2- 172.21.15.6:0/3702916665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1000b5740 0x7ff1000b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 >> 172.21.15.6:0/3702916665 conn(0x7ff100072d10 msgr2=0x7ff1000b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 shutdown_connections 2024-03-18T14:43:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:29.906+0000 7ff10820b640 1 -- 172.21.15.6:0/3702916665 wait complete. 2024-03-18T14:43:29.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T14:43:30.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.072+0000 7feeb9da9640 1 Processor -- start 2024-03-18T14:43:30.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.072+0000 7feeb9da9640 1 -- start start 2024-03-18T14:43:30.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.073+0000 7feeb9da9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40cd450 0x7feeb4168e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.073+0000 7feeb9da9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb40b1eb0 con 0x7feeb40b1a20 2024-03-18T14:43:30.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.073+0000 7feeb9da9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb40b2190 con 0x7feeb406cce0 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.073+0000 7feeb9da9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7feeb40b2ac0 con 0x7feeb40cd450 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.073+0000 7feeb2ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feeb40b1a20 0x7feeb40ceda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59872/0 (socket says 172.21.15.6:59872) 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.073+0000 7feeb2ffd640 1 -- 172.21.15.6:0/3804728443 learned_addr learned my addr 172.21.15.6:0/3804728443 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.073+0000 7feeb3fff640 1 --2- 172.21.15.6:0/3804728443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40cd450 0x7feeb4168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.074+0000 7feeb27fc640 1 -- 172.21.15.6:0/3804728443 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428281481 0 0) 0x7feeb40b1eb0 con 0x7feeb40b1a20 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.074+0000 7feeb27fc640 1 -- 172.21.15.6:0/3804728443 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee9c003520 con 0x7feeb40b1a20 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.074+0000 7feeb27fc640 1 -- 172.21.15.6:0/3804728443 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230369365 0 0) 0x7feeb40b2190 con 0x7feeb406cce0 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.074+0000 7feeb27fc640 1 -- 172.21.15.6:0/3804728443 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feeb40b1eb0 con 0x7feeb406cce0 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.074+0000 7feeb3fff640 1 -- 172.21.15.6:0/3804728443 >> v1:172.21.15.92:6789/0 conn(0x7feeb406cce0 legacy=0x7feeb406d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.074+0000 7feeb3fff640 1 -- 172.21.15.6:0/3804728443 >> v1:172.21.15.6:6789/0 conn(0x7feeb40b1a20 legacy=0x7feeb40ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.074+0000 7feeb3fff640 1 -- 172.21.15.6:0/3804728443 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb40b2470 con 0x7feeb40cd450 2024-03-18T14:43:30.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.075+0000 7feeb3fff640 1 --2- 172.21.15.6:0/3804728443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40cd450 0x7feeb4168e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feeac009dc0 tx=0x7feeac02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b90fac1683a40d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.075+0000 7feeb27fc640 1 -- 172.21.15.6:0/3804728443 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac039070 con 0x7feeb40cd450 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.075+0000 7feeb27fc640 1 -- 172.21.15.6:0/3804728443 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeac02ccd0 con 0x7feeb40cd450 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.075+0000 7feeb27fc640 1 -- 172.21.15.6:0/3804728443 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac034660 con 0x7feeb40cd450 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.076+0000 7feeb9da9640 1 -- 172.21.15.6:0/3804728443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40cd450 msgr2=0x7feeb4168e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.076+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3804728443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40cd450 0x7feeb4168e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feeac009dc0 tx=0x7feeac02bf20 comp rx=0 tx=0).stop 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.076+0000 7feeb9da9640 1 -- 172.21.15.6:0/3804728443 shutdown_connections 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.076+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3804728443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40cd450 0x7feeb4168e40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.076+0000 7feeb9da9640 1 -- 172.21.15.6:0/3804728443 >> 172.21.15.6:0/3804728443 conn(0x7feeb409ae20 msgr2=0x7feeb408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.076+0000 7feeb9da9640 1 -- 172.21.15.6:0/3804728443 shutdown_connections 2024-03-18T14:43:30.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.076+0000 7feeb9da9640 1 -- 172.21.15.6:0/3804728443 wait complete. 2024-03-18T14:43:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.077+0000 7feeb9da9640 1 Processor -- start 2024-03-18T14:43:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.077+0000 7feeb9da9640 1 -- start start 2024-03-18T14:43:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeb406cce0 0x7feeb4157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb40cd450 0x7feeb40b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40b7520 0x7feeb415d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feeb40d2960 con 0x7feeb406cce0 2024-03-18T14:43:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feeb40d27e0 con 0x7feeb40cd450 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7feeb4169380 con 0x7feeb40b7520 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeb406cce0 0x7feeb4157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeb406cce0 0x7feeb4157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34920/0 (socket says 172.21.15.6:34920) 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 -- 172.21.15.6:0/3706478543 learned_addr learned my addr 172.21.15.6:0/3706478543 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb2ffd640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb40cd450 0x7feeb40b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb3fff640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40b7520 0x7feeb415d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 -- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40b7520 msgr2=0x7feeb415d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40b7520 0x7feeb415d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 -- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb40cd450 msgr2=0x7feeb40b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb40cd450 0x7feeb40b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 -- 172.21.15.6:0/3706478543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb40b6ab0 con 0x7feeb406cce0 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb2ffd640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb40cd450 0x7feeb40b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb37fe640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeb406cce0 0x7feeb4157990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feea400c500 tx=0x7feea400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7fee9bfff640 1 -- 172.21.15.6:0/3706478543 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea40052d0 con 0x7feeb406cce0 2024-03-18T14:43:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7fee9bfff640 1 -- 172.21.15.6:0/3706478543 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea4005490 con 0x7feeb406cce0 2024-03-18T14:43:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb40b5060 con 0x7feeb406cce0 2024-03-18T14:43:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7fee9bfff640 1 -- 172.21.15.6:0/3706478543 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4012870 con 0x7feeb406cce0 2024-03-18T14:43:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.078+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb40b5340 con 0x7feeb406cce0 2024-03-18T14:43:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.080+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb40cbea0 con 0x7feeb406cce0 2024-03-18T14:43:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.080+0000 7fee9bfff640 1 -- 172.21.15.6:0/3706478543 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feea4012a10 con 0x7feeb406cce0 2024-03-18T14:43:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.080+0000 7fee9bfff640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feea006bbd0 0x7feea006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.080+0000 7fee9bfff640 1 -- 172.21.15.6:0/3706478543 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6611+0+0 (secure 0 0 0) 0x7feea408aa40 con 0x7feeb406cce0 2024-03-18T14:43:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.080+0000 7feeb2ffd640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feea006bbd0 0x7feea006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.081+0000 7feeb2ffd640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feea006bbd0 0x7feea006e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7feeb40b0e80 tx=0x7feea8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:30.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.085+0000 7fee9bfff640 1 -- 172.21.15.6:0/3706478543 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feea40591b0 con 0x7feeb406cce0 2024-03-18T14:43:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.266+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7feeb404f620 con 0x7feeb406cce0 2024-03-18T14:43:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.905+0000 7fee9bfff640 1 -- 172.21.15.6:0/3706478543 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) ==== 181+0+0 (secure 0 0 0) 0x7feea4058ab0 con 0x7feeb406cce0 2024-03-18T14:43:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feea006bbd0 msgr2=0x7feea006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feea006bbd0 0x7feea006e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7feeb40b0e80 tx=0x7feea8009040 comp rx=0 tx=0).stop 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeb406cce0 msgr2=0x7feeb4157990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeb406cce0 0x7feeb4157990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feea400c500 tx=0x7feea400c9c0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 shutdown_connections 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7feea006bbd0 0x7feea006e090 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feeb40b7520 0x7feeb415d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb40cd450 0x7feeb40b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 --2- 172.21.15.6:0/3706478543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feeb406cce0 0x7feeb4157990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.908+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 >> 172.21.15.6:0/3706478543 conn(0x7feeb409ae20 msgr2=0x7feeb40d1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.909+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 shutdown_connections 2024-03-18T14:43:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:30.909+0000 7feeb9da9640 1 -- 172.21.15.6:0/3706478543 wait complete. 2024-03-18T14:43:30.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T14:43:31.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.034+0000 7f60f7e36640 1 Processor -- start 2024-03-18T14:43:31.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.034+0000 7f60f7e36640 1 -- start start 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f7e36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60f00ccd70 0x7f60f00cd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f7e36640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60f006dff0 0x7f60f006e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f7e36640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f0072390 con 0x7f60f01424f0 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f7e36640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f60f01428d0 con 0x7f60f00ccd70 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f7e36640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f60f0072670 con 0x7f60f006dff0 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f53d1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60f00ccd70 0x7f60f00cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f53d1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60f00ccd70 0x7f60f00cd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34936/0 (socket says 172.21.15.6:34936) 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f53d1640 1 -- 172.21.15.6:0/3739922957 learned_addr learned my addr 172.21.15.6:0/3739922957 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f63d3640 1 --2- 172.21.15.6:0/3739922957 >> v2:172.21.15.92:3300/0 conn(0x7f60f006dff0 0x7f60f006e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f53d1640 1 -- 172.21.15.6:0/3739922957 >> v2:172.21.15.92:3300/0 conn(0x7f60f006dff0 msgr2=0x7f60f006e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f53d1640 1 --2- 172.21.15.6:0/3739922957 >> v2:172.21.15.92:3300/0 conn(0x7f60f006dff0 0x7f60f006e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f53d1640 1 -- 172.21.15.6:0/3739922957 >> v1:172.21.15.92:6789/0 conn(0x7f60f01424f0 legacy=0x7f60f00718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.035+0000 7f60f53d1640 1 -- 172.21.15.6:0/3739922957 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f0072950 con 0x7f60f00ccd70 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.036+0000 7f60f53d1640 1 --2- 172.21.15.6:0/3739922957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f00ccd70 0x7f60f00cd200 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f60f006f960 tx=0x7f60e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac7cc211b38a0e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.036+0000 7f60f4bd0640 1 -- 172.21.15.6:0/3739922957 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0034070 con 0x7f60f00ccd70 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.036+0000 7f60f4bd0640 1 -- 172.21.15.6:0/3739922957 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e0030860 con 0x7f60f00ccd70 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.036+0000 7f60f4bd0640 1 -- 172.21.15.6:0/3739922957 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0030c30 con 0x7f60f00ccd70 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 -- 172.21.15.6:0/3739922957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f00ccd70 msgr2=0x7f60f00cd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 --2- 172.21.15.6:0/3739922957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f00ccd70 0x7f60f00cd200 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f60f006f960 tx=0x7f60e002b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 -- 172.21.15.6:0/3739922957 shutdown_connections 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 --2- 172.21.15.6:0/3739922957 >> v2:172.21.15.92:3300/0 conn(0x7f60f006dff0 0x7f60f006e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 --2- 172.21.15.6:0/3739922957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f00ccd70 0x7f60f00cd200 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 -- 172.21.15.6:0/3739922957 >> 172.21.15.6:0/3739922957 conn(0x7f60f00b1500 msgr2=0x7f60f00b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:31.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 -- 172.21.15.6:0/3739922957 shutdown_connections 2024-03-18T14:43:31.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.037+0000 7f60f7e36640 1 -- 172.21.15.6:0/3739922957 wait complete. 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.038+0000 7f60f7e36640 1 Processor -- start 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.038+0000 7f60f7e36640 1 -- start start 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f7e36640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f006dff0 0x7f60f00b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f7e36640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60f00ccd70 0x7f60f00b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f7e36640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60f01424f0 0x7f60f01f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f7e36640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60f0072000 con 0x7f60f006dff0 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f7e36640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f60f00cefd0 con 0x7f60f00ccd70 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f7e36640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f60f006e940 con 0x7f60f01424f0 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f006dff0 0x7f60f00b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f006dff0 0x7f60f00b4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34952/0 (socket says 172.21.15.6:34952) 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 -- 172.21.15.6:0/1854875735 learned_addr learned my addr 172.21.15.6:0/1854875735 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f53d1640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60f00ccd70 0x7f60f00b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f63d3640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60f01424f0 0x7f60f01f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 -- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60f01424f0 msgr2=0x7f60f01f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60f01424f0 0x7f60f01f3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 -- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60f00ccd70 msgr2=0x7f60f00b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60f00ccd70 0x7f60f00b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.039+0000 7f60f5bd2640 1 -- 172.21.15.6:0/1854875735 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f00b2f90 con 0x7f60f006dff0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.040+0000 7f60f5bd2640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f006dff0 0x7f60f00b4030 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c3c0 tx=0x7f60e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.040+0000 7f60e67fc640 1 -- 172.21.15.6:0/1854875735 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e80052d0 con 0x7f60f006dff0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.040+0000 7f60e67fc640 1 -- 172.21.15.6:0/1854875735 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e8005490 con 0x7f60f006dff0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.040+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f00b3270 con 0x7f60f006dff0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.040+0000 7f60e67fc640 1 -- 172.21.15.6:0/1854875735 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8007260 con 0x7f60f006dff0 2024-03-18T14:43:31.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.040+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f00b3830 con 0x7f60f006dff0 2024-03-18T14:43:31.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.041+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f004f620 con 0x7f60f006dff0 2024-03-18T14:43:31.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.041+0000 7f60e67fc640 1 -- 172.21.15.6:0/1854875735 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f60e8007400 con 0x7f60f006dff0 2024-03-18T14:43:31.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.042+0000 7f60e67fc640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f60d406bbd0 0x7f60d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:31.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.042+0000 7f60e67fc640 1 -- 172.21.15.6:0/1854875735 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6173+0+0 (secure 0 0 0) 0x7f60e8089fe0 con 0x7f60f006dff0 2024-03-18T14:43:31.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.042+0000 7f60f53d1640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f60d406bbd0 0x7f60d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:31.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.042+0000 7f60f53d1640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f60d406bbd0 0x7f60d406e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f60e0036000 tx=0x7f60e0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:31.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.046+0000 7f60e67fc640 1 -- 172.21.15.6:0/1854875735 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f60e8058340 con 0x7f60f006dff0 2024-03-18T14:43:31.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.229+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f60f00b0e40 con 0x7f60f006dff0 2024-03-18T14:43:31.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.910+0000 7f60e67fc640 1 -- 172.21.15.6:0/1854875735 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) ==== 169+0+0 (secure 0 0 0) 0x7f60e805c3e0 con 0x7f60f006dff0 2024-03-18T14:43:31.912 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.914+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f60d406bbd0 msgr2=0x7f60d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.914+0000 7f60f7e36640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f60d406bbd0 0x7f60d406e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f60e0036000 tx=0x7f60e0008030 comp rx=0 tx=0).stop 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.914+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f006dff0 msgr2=0x7f60f00b4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.914+0000 7f60f7e36640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f006dff0 0x7f60f00b4030 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c3c0 tx=0x7f60e800c880 comp rx=0 tx=0).stop 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.915+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 shutdown_connections 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.915+0000 7f60f7e36640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f60d406bbd0 0x7f60d406e090 secure :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f60e0036000 tx=0x7f60e0008030 comp rx=0 tx=0).stop 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.915+0000 7f60f7e36640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60f01424f0 0x7f60f01f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.915+0000 7f60f7e36640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60f00ccd70 0x7f60f00b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.915+0000 7f60f7e36640 1 --2- 172.21.15.6:0/1854875735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60f006dff0 0x7f60f00b4030 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:31.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.915+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 >> 172.21.15.6:0/1854875735 conn(0x7f60f00b1500 msgr2=0x7f60f00cdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:31.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.916+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 shutdown_connections 2024-03-18T14:43:31.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:31.916+0000 7f60f7e36640 1 -- 172.21.15.6:0/1854875735 wait complete. 2024-03-18T14:43:31.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2024-03-18T14:43:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.041+0000 7f80246ff640 1 Processor -- start 2024-03-18T14:43:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.041+0000 7f80246ff640 1 -- start start 2024-03-18T14:43:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f80246ff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f801c15e0b0 0x7f801c0b95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f80246ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f801c06c730 0x7f801c06cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f80246ff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f801c0d22c0 con 0x7f801c0d1de0 2024-03-18T14:43:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f80246ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f801c06d0a0 con 0x7f801c06c730 2024-03-18T14:43:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f80246ff640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f801c0d25a0 con 0x7f801c15e0b0 2024-03-18T14:43:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f8022c9c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f801c06c730 0x7f801c06cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f8022c9c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f801c06c730 0x7f801c06cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34958/0 (socket says 172.21.15.6:34958) 2024-03-18T14:43:32.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.042+0000 7f8022c9c640 1 -- 172.21.15.6:0/1029354959 learned_addr learned my addr 172.21.15.6:0/1029354959 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:32.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.043+0000 7f802249b640 1 --2- 172.21.15.6:0/1029354959 >> v2:172.21.15.92:3300/0 conn(0x7f801c15e0b0 0x7f801c0b95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:32.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.043+0000 7f8022c9c640 1 -- 172.21.15.6:0/1029354959 >> v2:172.21.15.92:3300/0 conn(0x7f801c15e0b0 msgr2=0x7f801c0b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:32.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.043+0000 7f8022c9c640 1 --2- 172.21.15.6:0/1029354959 >> v2:172.21.15.92:3300/0 conn(0x7f801c15e0b0 0x7f801c0b95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.043+0000 7f8022c9c640 1 -- 172.21.15.6:0/1029354959 >> v1:172.21.15.92:6789/0 conn(0x7f801c0d1de0 legacy=0x7f801c0d3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:32.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.043+0000 7f8022c9c640 1 -- 172.21.15.6:0/1029354959 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801c0d2880 con 0x7f801c06c730 2024-03-18T14:43:32.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f8022c9c640 1 --2- 172.21.15.6:0/1029354959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c06c730 0x7f801c06cb60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f801c0d1690 tx=0x7f801802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3373ba5d17d22df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f8021499640 1 -- 172.21.15.6:0/1029354959 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801802cbb0 con 0x7f801c06c730 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f8021499640 1 -- 172.21.15.6:0/1029354959 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f801802cd50 con 0x7f801c06c730 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f8021499640 1 -- 172.21.15.6:0/1029354959 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018034610 con 0x7f801c06c730 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 -- 172.21.15.6:0/1029354959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c06c730 msgr2=0x7f801c06cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 --2- 172.21.15.6:0/1029354959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c06c730 0x7f801c06cb60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f801c0d1690 tx=0x7f801802c010 comp rx=0 tx=0).stop 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 -- 172.21.15.6:0/1029354959 shutdown_connections 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 --2- 172.21.15.6:0/1029354959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c06c730 0x7f801c06cb60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 --2- 172.21.15.6:0/1029354959 >> v2:172.21.15.92:3300/0 conn(0x7f801c15e0b0 0x7f801c0b95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 -- 172.21.15.6:0/1029354959 >> 172.21.15.6:0/1029354959 conn(0x7f801c072ce0 msgr2=0x7f801c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 -- 172.21.15.6:0/1029354959 shutdown_connections 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.044+0000 7f80246ff640 1 -- 172.21.15.6:0/1029354959 wait complete. 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 Processor -- start 2024-03-18T14:43:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 -- start start 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f801c06c730 0x7f801c0b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f801c0d1de0 0x7f801c0b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c15e0b0 0x7f801c1583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f801c0b9b30 con 0x7f801c15e0b0 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f801c0d2e40 con 0x7f801c06c730 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f80246ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f801c1588e0 con 0x7f801c0d1de0 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f8022c9c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c15e0b0 0x7f801c1583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f8022c9c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c15e0b0 0x7f801c1583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34962/0 (socket says 172.21.15.6:34962) 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f8022c9c640 1 -- 172.21.15.6:0/4063462120 learned_addr learned my addr 172.21.15.6:0/4063462120 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.046+0000 7f802249b640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f801c06c730 0x7f801c0b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8021c9a640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f801c0d1de0 0x7f801c0b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8022c9c640 1 -- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f801c0d1de0 msgr2=0x7f801c0b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8022c9c640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f801c0d1de0 0x7f801c0b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8022c9c640 1 -- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f801c06c730 msgr2=0x7f801c0b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8022c9c640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f801c06c730 0x7f801c0b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8022c9c640 1 -- 172.21.15.6:0/4063462120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801c0b5a10 con 0x7f801c15e0b0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8021c9a640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f801c0d1de0 0x7f801c0b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f802249b640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f801c06c730 0x7f801c0b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8022c9c640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c15e0b0 0x7f801c1583a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8018009f40 tx=0x7f8018004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8012ffd640 1 -- 172.21.15.6:0/4063462120 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018043020 con 0x7f801c15e0b0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8012ffd640 1 -- 172.21.15.6:0/4063462120 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f801803e450 con 0x7f801c15e0b0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f801c0b3c70 con 0x7f801c15e0b0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f8012ffd640 1 -- 172.21.15.6:0/4063462120 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018009510 con 0x7f801c15e0b0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.047+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f801c0b4230 con 0x7f801c15e0b0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.048+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f801c08afe0 con 0x7f801c15e0b0 2024-03-18T14:43:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.049+0000 7f8012ffd640 1 -- 172.21.15.6:0/4063462120 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8018046020 con 0x7f801c15e0b0 2024-03-18T14:43:32.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.049+0000 7f8012ffd640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f800006bbd0 0x7f800006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:32.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.049+0000 7f8012ffd640 1 -- 172.21.15.6:0/4063462120 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5730+0+0 (secure 0 0 0) 0x7f80180ad550 con 0x7f801c15e0b0 2024-03-18T14:43:32.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.049+0000 7f802249b640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f800006bbd0 0x7f800006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:32.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.050+0000 7f802249b640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f800006bbd0 0x7f800006e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f801c1ea490 tx=0x7f8014008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:32.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.053+0000 7f8012ffd640 1 -- 172.21.15.6:0/4063462120 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f801804d020 con 0x7f801c15e0b0 2024-03-18T14:43:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.236+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) -- 0x7f801c04f620 con 0x7f801c15e0b0 2024-03-18T14:43:32.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.912+0000 7f8012ffd640 1 -- 172.21.15.6:0/4063462120 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v48) ==== 160+0+0 (secure 0 0 0) 0x7f801807c1d0 con 0x7f801c15e0b0 2024-03-18T14:43:32.913 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'data_new' removed 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f800006bbd0 msgr2=0x7f800006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f800006bbd0 0x7f800006e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f801c1ea490 tx=0x7f8014008040 comp rx=0 tx=0).stop 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c15e0b0 msgr2=0x7f801c1583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c15e0b0 0x7f801c1583a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8018009f40 tx=0x7f8018004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 shutdown_connections 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f800006bbd0 0x7f800006e090 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f801c15e0b0 0x7f801c1583a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f801c0d1de0 0x7f801c0b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 --2- 172.21.15.6:0/4063462120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f801c06c730 0x7f801c0b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:32.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.914+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 >> 172.21.15.6:0/4063462120 conn(0x7f801c072ce0 msgr2=0x7f801c0ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:32.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.915+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 shutdown_connections 2024-03-18T14:43:32.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:32.915+0000 7f80246ff640 1 -- 172.21.15.6:0/4063462120 wait complete. 2024-03-18T14:43:32.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T14:43:33.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.039+0000 7f26c68ff640 1 Processor -- start 2024-03-18T14:43:33.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.039+0000 7f26c68ff640 1 -- start start 2024-03-18T14:43:33.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26c68ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f26c00d2e50 0x7f26c00b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26c68ff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26c015e3b0 con 0x7f26c00b5c80 2024-03-18T14:43:33.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26c68ff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26c00b8470 con 0x7f26c015dfd0 2024-03-18T14:43:33.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26c68ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f26c00b8750 con 0x7f26c00d2e50 2024-03-18T14:43:33.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26bf7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f26c00d2e50 0x7f26c00b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26c4e9c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f26c00b5c80 0x7f26c00b6df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59880/0 (socket says 172.21.15.6:59880) 2024-03-18T14:43:33.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26c4e9c640 1 -- 172.21.15.6:0/3314611789 learned_addr learned my addr 172.21.15.6:0/3314611789 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:33.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26bf7fe640 1 -- 172.21.15.6:0/3314611789 >> v1:172.21.15.92:6789/0 conn(0x7f26c015dfd0 legacy=0x7f26c00b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:33.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26bf7fe640 1 -- 172.21.15.6:0/3314611789 >> v1:172.21.15.6:6789/0 conn(0x7f26c00b5c80 legacy=0x7f26c00b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.040+0000 7f26bf7fe640 1 -- 172.21.15.6:0/3314611789 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c00b8a30 con 0x7f26c00d2e50 2024-03-18T14:43:33.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.041+0000 7f26bf7fe640 1 --2- 172.21.15.6:0/3314611789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c00b5740 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f26b40097a0 tx=0x7f26b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191264f028f89a64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.041+0000 7f26beffd640 1 -- 172.21.15.6:0/3314611789 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4035020 con 0x7f26c00d2e50 2024-03-18T14:43:33.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.041+0000 7f26beffd640 1 -- 172.21.15.6:0/3314611789 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b4031830 con 0x7f26c00d2e50 2024-03-18T14:43:33.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.041+0000 7f26beffd640 1 -- 172.21.15.6:0/3314611789 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4031c00 con 0x7f26c00d2e50 2024-03-18T14:43:33.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.042+0000 7f26c68ff640 1 -- 172.21.15.6:0/3314611789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 msgr2=0x7f26c00b5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.042+0000 7f26c68ff640 1 --2- 172.21.15.6:0/3314611789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c00b5740 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f26b40097a0 tx=0x7f26b402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.042+0000 7f26c68ff640 1 -- 172.21.15.6:0/3314611789 shutdown_connections 2024-03-18T14:43:33.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.042+0000 7f26c68ff640 1 --2- 172.21.15.6:0/3314611789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c00b5740 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.042+0000 7f26c68ff640 1 -- 172.21.15.6:0/3314611789 >> 172.21.15.6:0/3314611789 conn(0x7f26c0072d10 msgr2=0x7f26c0073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:33.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.042+0000 7f26c68ff640 1 -- 172.21.15.6:0/3314611789 shutdown_connections 2024-03-18T14:43:33.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.042+0000 7f26c68ff640 1 -- 172.21.15.6:0/3314611789 wait complete. 2024-03-18T14:43:33.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.043+0000 7f26c68ff640 1 Processor -- start 2024-03-18T14:43:33.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.043+0000 7f26c68ff640 1 -- start start 2024-03-18T14:43:33.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26c68ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c00b5c80 0x7f26c0153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26c68ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c01566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26c68ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c015dfd0 0x7f26c0156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26c68ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f26c00b7680 con 0x7f26c00d2e50 2024-03-18T14:43:33.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26c68ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f26c00b7500 con 0x7f26c00b5c80 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26c68ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f26c00b7800 con 0x7f26c015dfd0 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c01566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c01566e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34974/0 (socket says 172.21.15.6:34974) 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 -- 172.21.15.6:0/1528013269 learned_addr learned my addr 172.21.15.6:0/1528013269 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26c4e9c640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c015dfd0 0x7f26c0156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bffff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c00b5c80 0x7f26c0153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 -- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c015dfd0 msgr2=0x7f26c0156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c015dfd0 0x7f26c0156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 -- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c00b5c80 msgr2=0x7f26c0153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c00b5c80 0x7f26c0153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 -- 172.21.15.6:0/1528013269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c0160200 con 0x7f26c00d2e50 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.044+0000 7f26bf7fe640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c01566e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f26b402b910 tx=0x7f26b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.045+0000 7f26bcff9640 1 -- 172.21.15.6:0/1528013269 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4041070 con 0x7f26c00d2e50 2024-03-18T14:43:33.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.045+0000 7f26bcff9640 1 -- 172.21.15.6:0/1528013269 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b403a970 con 0x7f26c00d2e50 2024-03-18T14:43:33.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.045+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c01604e0 con 0x7f26c00d2e50 2024-03-18T14:43:33.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.045+0000 7f26bcff9640 1 -- 172.21.15.6:0/1528013269 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4035040 con 0x7f26c00d2e50 2024-03-18T14:43:33.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.045+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c0160aa0 con 0x7f26c00d2e50 2024-03-18T14:43:33.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.046+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c00751b0 con 0x7f26c00d2e50 2024-03-18T14:43:33.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.046+0000 7f26bcff9640 1 -- 172.21.15.6:0/1528013269 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f26b404c060 con 0x7f26c00d2e50 2024-03-18T14:43:33.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.047+0000 7f26bcff9640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f26ac06bc00 0x7f26ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.047+0000 7f26bcff9640 1 -- 172.21.15.6:0/1528013269 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7f26b40ac3a0 con 0x7f26c00d2e50 2024-03-18T14:43:33.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.047+0000 7f26bffff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f26ac06bc00 0x7f26ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.047+0000 7f26bffff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f26ac06bc00 0x7f26ac06e0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f26b0006fd0 tx=0x7f26b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.051+0000 7f26bcff9640 1 -- 172.21.15.6:0/1528013269 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f26b407b000 con 0x7f26c00d2e50 2024-03-18T14:43:33.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.234+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f26c006b910 con 0x7f26c00d2e50 2024-03-18T14:43:33.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.237+0000 7f26bcff9640 1 -- 172.21.15.6:0/1528013269 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f26b407a900 con 0x7f26c00d2e50 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f26ac06bc00 msgr2=0x7f26ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f26ac06bc00 0x7f26ac06e0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f26b0006fd0 tx=0x7f26b0008040 comp rx=0 tx=0).stop 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 msgr2=0x7f26c01566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c01566e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f26b402b910 tx=0x7f26b40046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 shutdown_connections 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f26ac06bc00 0x7f26ac06e0c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c015dfd0 0x7f26c0156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c00d2e50 0x7f26c01566e0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 --2- 172.21.15.6:0/1528013269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c00b5c80 0x7f26c0153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 >> 172.21.15.6:0/1528013269 conn(0x7f26c0072d10 msgr2=0x7f26c00b4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 shutdown_connections 2024-03-18T14:43:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.239+0000 7f26c68ff640 1 -- 172.21.15.6:0/1528013269 wait complete. 2024-03-18T14:43:33.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-18T14:43:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.351+0000 7f7a8709a640 1 Processor -- start 2024-03-18T14:43:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.351+0000 7f7a8709a640 1 -- start start 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.352+0000 7f7a8709a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a800d1f60 0x7f7a800cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.352+0000 7f7a8709a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a800cc870 con 0x7f7a8015bd40 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.352+0000 7f7a8709a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a800d0500 con 0x7f7a800cfcd0 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.352+0000 7f7a8709a640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7a800d07e0 con 0x7f7a800d1f60 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.352+0000 7f7a7ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a800d1f60 0x7f7a800cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.352+0000 7f7a84e36640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7a8015bd40 0x7f7a800d1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59886/0 (socket says 172.21.15.6:59886) 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.352+0000 7f7a84e36640 1 -- 172.21.15.6:0/2760297478 learned_addr learned my addr 172.21.15.6:0/2760297478 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.353+0000 7f7a7f7fe640 1 -- 172.21.15.6:0/2760297478 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1451033857 0 0) 0x7f7a800cc870 con 0x7f7a8015bd40 2024-03-18T14:43:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.353+0000 7f7a7f7fe640 1 -- 172.21.15.6:0/2760297478 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a6c003520 con 0x7f7a8015bd40 2024-03-18T14:43:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.353+0000 7f7a7ffff640 1 -- 172.21.15.6:0/2760297478 >> v1:172.21.15.92:6789/0 conn(0x7f7a800cfcd0 legacy=0x7f7a800cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.353+0000 7f7a7ffff640 1 -- 172.21.15.6:0/2760297478 >> v1:172.21.15.6:6789/0 conn(0x7f7a8015bd40 legacy=0x7f7a800d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.353+0000 7f7a7ffff640 1 -- 172.21.15.6:0/2760297478 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a800ccb10 con 0x7f7a800d1f60 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.353+0000 7f7a7ffff640 1 --2- 172.21.15.6:0/2760297478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a800d1f60 0x7f7a800cf790 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7a80067f50 tx=0x7f7a7002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcd480e7537940b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a7f7fe640 1 -- 172.21.15.6:0/2760297478 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70039070 con 0x7f7a800d1f60 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a7f7fe640 1 -- 172.21.15.6:0/2760297478 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a7002cbc0 con 0x7f7a800d1f60 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a7f7fe640 1 -- 172.21.15.6:0/2760297478 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70034440 con 0x7f7a800d1f60 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a8709a640 1 -- 172.21.15.6:0/2760297478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a800d1f60 msgr2=0x7f7a800cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a8709a640 1 --2- 172.21.15.6:0/2760297478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a800d1f60 0x7f7a800cf790 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7a80067f50 tx=0x7f7a7002be10 comp rx=0 tx=0).stop 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a8709a640 1 -- 172.21.15.6:0/2760297478 shutdown_connections 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a8709a640 1 --2- 172.21.15.6:0/2760297478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a800d1f60 0x7f7a800cf790 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a8709a640 1 -- 172.21.15.6:0/2760297478 >> 172.21.15.6:0/2760297478 conn(0x7f7a8009ae20 msgr2=0x7f7a8008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a8709a640 1 -- 172.21.15.6:0/2760297478 shutdown_connections 2024-03-18T14:43:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.354+0000 7f7a8709a640 1 -- 172.21.15.6:0/2760297478 wait complete. 2024-03-18T14:43:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.355+0000 7f7a8709a640 1 Processor -- start 2024-03-18T14:43:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.355+0000 7f7a8709a640 1 -- start start 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a8709a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a800cfcd0 0x7f7a801f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a8709a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a800d1f60 0x7f7a800ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a8709a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8015bd40 0x7f7a80151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a8709a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7a800d0280 con 0x7f7a8015bd40 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a8709a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7a800d0100 con 0x7f7a800cfcd0 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a8709a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8015c120 con 0x7f7a800d1f60 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a85637640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8015bd40 0x7f7a80151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a85637640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8015bd40 0x7f7a80151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35002/0 (socket says 172.21.15.6:35002) 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.356+0000 7f7a85637640 1 -- 172.21.15.6:0/420705693 learned_addr learned my addr 172.21.15.6:0/420705693 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a84e36640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a800cfcd0 0x7f7a801f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a7ffff640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a800d1f60 0x7f7a800ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a85637640 1 -- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a800d1f60 msgr2=0x7f7a800ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a85637640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a800d1f60 0x7f7a800ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a85637640 1 -- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a800cfcd0 msgr2=0x7f7a801f4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a85637640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a800cfcd0 0x7f7a801f4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a85637640 1 -- 172.21.15.6:0/420705693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a800afb80 con 0x7f7a8015bd40 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a85637640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8015bd40 0x7f7a80151820 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c390 tx=0x7f7a6800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a7d7fa640 1 -- 172.21.15.6:0/420705693 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68017070 con 0x7f7a8015bd40 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a7d7fa640 1 -- 172.21.15.6:0/420705693 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a680035f0 con 0x7f7a8015bd40 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a7d7fa640 1 -- 172.21.15.6:0/420705693 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a680039e0 con 0x7f7a8015bd40 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a800afe60 con 0x7f7a8015bd40 2024-03-18T14:43:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.357+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a800c8e70 con 0x7f7a8015bd40 2024-03-18T14:43:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.358+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8009fa90 con 0x7f7a8015bd40 2024-03-18T14:43:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.358+0000 7f7a7d7fa640 1 -- 172.21.15.6:0/420705693 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7a6801a070 con 0x7f7a8015bd40 2024-03-18T14:43:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.359+0000 7f7a7d7fa640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7a6006bbd0 0x7f7a6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.359+0000 7f7a7d7fa640 1 -- 172.21.15.6:0/420705693 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7f7a68089ce0 con 0x7f7a8015bd40 2024-03-18T14:43:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.359+0000 7f7a84e36640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7a6006bbd0 0x7f7a6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.360+0000 7f7a84e36640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7a6006bbd0 0x7f7a6006e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7a78005880 tx=0x7f7a78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:33.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.363+0000 7f7a7d7fa640 1 -- 172.21.15.6:0/420705693 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7a68058940 con 0x7f7a8015bd40 2024-03-18T14:43:33.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.627+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7f7a8004f620 con 0x7f7a8015bd40 2024-03-18T14:43:33.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.923+0000 7f7a7d7fa640 1 -- 172.21.15.6:0/420705693 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v138) ==== 147+0+0 (secure 0 0 0) 0x7f7a68058240 con 0x7f7a8015bd40 2024-03-18T14:43:33.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7a6006bbd0 msgr2=0x7f7a6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7a6006bbd0 0x7f7a6006e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7a78005880 tx=0x7f7a78009040 comp rx=0 tx=0).stop 2024-03-18T14:43:33.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8015bd40 msgr2=0x7f7a80151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:33.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8015bd40 0x7f7a80151820 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c390 tx=0x7f7a6800c850 comp rx=0 tx=0).stop 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 shutdown_connections 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7a6006bbd0 0x7f7a6006e090 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8015bd40 0x7f7a80151820 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a800d1f60 0x7f7a800ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 --2- 172.21.15.6:0/420705693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a800cfcd0 0x7f7a801f4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 >> 172.21.15.6:0/420705693 conn(0x7f7a8009ae20 msgr2=0x7f7a800d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 shutdown_connections 2024-03-18T14:43:33.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:33.926+0000 7f7a8709a640 1 -- 172.21.15.6:0/420705693 wait complete. 2024-03-18T14:43:33.939 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2024-03-18T14:43:33.939 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-18T14:43:33.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:34.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.042+0000 7f52887f3640 1 Processor -- start 2024-03-18T14:43:34.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.042+0000 7f52887f3640 1 -- start start 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f52887f3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f528006e9c0 0x7f528015be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f52887f3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52800b2dd0 con 0x7f528015bab0 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f52887f3640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52800b30b0 con 0x7f528006dbb0 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f52887f3640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f52800b3390 con 0x7f528006e9c0 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f5285d8e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f528006e9c0 0x7f528015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f528658f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f528015bab0 0x7f528006e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56960/0 (socket says 172.21.15.6:56960) 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f528658f640 1 -- 172.21.15.6:0/2526781524 learned_addr learned my addr 172.21.15.6:0/2526781524 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f5285d8e640 1 -- 172.21.15.6:0/2526781524 >> v1:172.21.15.92:6789/0 conn(0x7f528006dbb0 legacy=0x7f52800b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f5285d8e640 1 -- 172.21.15.6:0/2526781524 >> v1:172.21.15.6:6789/0 conn(0x7f528015bab0 legacy=0x7f528006e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f528558d640 1 -- 172.21.15.6:0/2526781524 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3020909446 0 0) 0x7f52800b2dd0 con 0x7f528015bab0 2024-03-18T14:43:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.043+0000 7f5285d8e640 1 -- 172.21.15.6:0/2526781524 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52800b2a80 con 0x7f528006e9c0 2024-03-18T14:43:34.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.044+0000 7f5285d8e640 1 --2- 172.21.15.6:0/2526781524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f52740097a0 tx=0x7f527402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc6f5a73738cb99 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.044+0000 7f528558d640 1 -- 172.21.15.6:0/2526781524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274034070 con 0x7f528006e9c0 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.044+0000 7f528558d640 1 -- 172.21.15.6:0/2526781524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5274030610 con 0x7f528006e9c0 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.044+0000 7f528558d640 1 -- 172.21.15.6:0/2526781524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52740309e0 con 0x7f528006e9c0 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.045+0000 7f52887f3640 1 -- 172.21.15.6:0/2526781524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 msgr2=0x7f528015be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.045+0000 7f52887f3640 1 --2- 172.21.15.6:0/2526781524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f52740097a0 tx=0x7f527402b960 comp rx=0 tx=0).stop 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.045+0000 7f52887f3640 1 -- 172.21.15.6:0/2526781524 shutdown_connections 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.045+0000 7f52887f3640 1 --2- 172.21.15.6:0/2526781524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.045+0000 7f52887f3640 1 -- 172.21.15.6:0/2526781524 >> 172.21.15.6:0/2526781524 conn(0x7f52800b1470 msgr2=0x7f52800b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.045+0000 7f52887f3640 1 -- 172.21.15.6:0/2526781524 shutdown_connections 2024-03-18T14:43:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.045+0000 7f52887f3640 1 -- 172.21.15.6:0/2526781524 wait complete. 2024-03-18T14:43:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.046+0000 7f52887f3640 1 Processor -- start 2024-03-18T14:43:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.046+0000 7f52887f3640 1 -- start start 2024-03-18T14:43:34.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f52887f3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f528006dbb0 0x7f5280148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f52887f3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f52887f3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f528015bab0 0x7f52800cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f52887f3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f52801606f0 con 0x7f528006e9c0 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f52887f3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5280160570 con 0x7f528006dbb0 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f52887f3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f52800b42a0 con 0x7f528015bab0 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57268/0 (socket says 172.21.15.6:57268) 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 -- 172.21.15.6:0/2725671785 learned_addr learned my addr 172.21.15.6:0/2725671785 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f528658f640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f528006dbb0 0x7f5280148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5286d90640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f528015bab0 0x7f52800cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 -- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f528015bab0 msgr2=0x7f52800cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f528015bab0 0x7f52800cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 -- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f528006dbb0 msgr2=0x7f5280148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f528006dbb0 0x7f5280148630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 -- 172.21.15.6:0/2725671785 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528015c800 con 0x7f528006e9c0 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.047+0000 7f5285d8e640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f527402bf70 tx=0x7f52740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.048+0000 7f527effd640 1 -- 172.21.15.6:0/2725671785 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274048070 con 0x7f528006e9c0 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.048+0000 7f527effd640 1 -- 172.21.15.6:0/2725671785 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5274039a10 con 0x7f528006e9c0 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.048+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528015cae0 con 0x7f528006e9c0 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.048+0000 7f527effd640 1 -- 172.21.15.6:0/2725671785 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274034040 con 0x7f528006e9c0 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.048+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f528015d0a0 con 0x7f528006e9c0 2024-03-18T14:43:34.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.049+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52800ca7c0 con 0x7f528006e9c0 2024-03-18T14:43:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.049+0000 7f527effd640 1 -- 172.21.15.6:0/2725671785 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f527404b070 con 0x7f528006e9c0 2024-03-18T14:43:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.049+0000 7f527effd640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.049+0000 7f527effd640 1 -- 172.21.15.6:0/2725671785 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7f52740ac5d0 con 0x7f528006e9c0 2024-03-18T14:43:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.050+0000 7f528658f640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.050+0000 7f528658f640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f526c06bbd0 0x7f526c06e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5278000f60 tx=0x7f5278007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.054+0000 7f527effd640 1 -- 172.21.15.6:0/2725671785 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f527407b2c0 con 0x7f528006e9c0 2024-03-18T14:43:34.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.237+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5280073310 con 0x7f528006e9c0 2024-03-18T14:43:34.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.238+0000 7f527effd640 1 -- 172.21.15.6:0/2725671785 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+13195 (secure 0 0 0) 0x7f527407abc0 con 0x7f528006e9c0 2024-03-18T14:43:34.239 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:34.240 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:32.911055+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6834/2677248951":"2024-03-19T14:43:28.898266+0000","172.21.15.6:6832/2677248951":"2024-03-19T14:43:28.898266+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:34.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f526c06bbd0 msgr2=0x7f526c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f526c06bbd0 0x7f526c06e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5278000f60 tx=0x7f5278007c00 comp rx=0 tx=0).stop 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 msgr2=0x7f528015be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f527402bf70 tx=0x7f52740046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 shutdown_connections 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f528015bab0 0x7f52800cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f528006e9c0 0x7f528015be90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 --2- 172.21.15.6:0/2725671785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f528006dbb0 0x7f5280148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 >> 172.21.15.6:0/2725671785 conn(0x7f52800b1470 msgr2=0x7f52800ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 shutdown_connections 2024-03-18T14:43:34.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.240+0000 7f52887f3640 1 -- 172.21.15.6:0/2725671785 wait complete. 2024-03-18T14:43:34.252 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:34.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:34.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.356+0000 7ff2302f8640 1 Processor -- start 2024-03-18T14:43:34.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.356+0000 7ff2302f8640 1 -- start start 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff2302f8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 0x7ff22806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff2302f8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2280ce040 con 0x7ff22806b700 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff2302f8640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2280d0870 con 0x7ff22815e030 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff2302f8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff22815e410 con 0x7ff2280afac0 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22e895640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff22806b700 0x7ff2280cb010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56972/0 (socket says 172.21.15.6:56972) 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22e895640 1 -- 172.21.15.6:0/3394301299 learned_addr learned my addr 172.21.15.6:0/3394301299 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d893640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 0x7ff22806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737316125 0 0) 0x7ff220001490 con 0x7ff22806b700 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff214003520 con 0x7ff22806b700 2024-03-18T14:43:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3816810429 0 0) 0x7ff2280d0870 con 0x7ff22815e030 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff220001490 con 0x7ff22815e030 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 566497791 0 0) 0x7ff214003520 con 0x7ff22806b700 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 msgr2=0x7ff22806b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 0x7ff22806b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 >> v1:172.21.15.92:6789/0 conn(0x7ff22815e030 legacy=0x7ff2280cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2280d0b50 con 0x7ff22806b700 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7ff220001bc0 con 0x7ff22806b700 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.357+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 >> v1:172.21.15.6:6789/0 conn(0x7ff22806b700 legacy=0x7ff2280cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d092640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 0x7ff214003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d092640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2140040d0 0x7ff214004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d092640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff214004d40 0x7ff2140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff22815e410 con 0x7ff2140040d0 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff21400bb50 con 0x7ff214004d40 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff21400bd20 con 0x7ff2280afac0 2024-03-18T14:43:34.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d893640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2140040d0 0x7ff214004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22e094640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 0x7ff214003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22e895640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff214004d40 0x7ff2140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d893640 1 -- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 msgr2=0x7ff214003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d893640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2280afac0 0x7ff214003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d893640 1 -- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff214004d40 msgr2=0x7ff2140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d893640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff214004d40 0x7ff2140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.358+0000 7ff22d893640 1 -- 172.21.15.6:0/3394301299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff214003520 con 0x7ff2140040d0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff22d893640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2140040d0 0x7ff214004800 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff218009ac0 tx=0x7ff214005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a24d57087def28c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218039070 con 0x7ff2140040d0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff21802c6b0 con 0x7ff2140040d0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff22d092640 1 -- 172.21.15.6:0/3394301299 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218009690 con 0x7ff2140040d0 2024-03-18T14:43:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff2302f8640 1 -- 172.21.15.6:0/3394301299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2140040d0 msgr2=0x7ff214004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff2302f8640 1 --2- 172.21.15.6:0/3394301299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2140040d0 0x7ff214004800 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff218009ac0 tx=0x7ff214005830 comp rx=0 tx=0).stop 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff22e895640 1 -- 172.21.15.6:0/3394301299 reap_dead start 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff2302f8640 1 -- 172.21.15.6:0/3394301299 shutdown_connections 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff2302f8640 1 -- 172.21.15.6:0/3394301299 >> 172.21.15.6:0/3394301299 conn(0x7ff228099a50 msgr2=0x7ff228099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.359+0000 7ff2302f8640 1 -- 172.21.15.6:0/3394301299 shutdown_connections 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.360+0000 7ff2302f8640 1 -- 172.21.15.6:0/3394301299 wait complete. 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 Processor -- start 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 -- start start 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2140040d0 0x7ff2280c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2280c51a0 0x7ff22815b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22815b7a0 0x7ff2280c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff2280d11a0 con 0x7ff2280c51a0 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff2280d1630 con 0x7ff2140040d0 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.361+0000 7ff2302f8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff2281f4750 con 0x7ff22815b7a0 2024-03-18T14:43:34.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2280c51a0 0x7ff22815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2280c51a0 0x7ff22815b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57292/0 (socket says 172.21.15.6:57292) 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 -- 172.21.15.6:0/45652984 learned_addr learned my addr 172.21.15.6:0/45652984 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22e094640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2140040d0 0x7ff2280c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 -- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22815b7a0 msgr2=0x7ff2280c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22e895640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22815b7a0 0x7ff2280c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22815b7a0 0x7ff2280c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 -- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2140040d0 msgr2=0x7ff2280c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2140040d0 0x7ff2280c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 -- 172.21.15.6:0/45652984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2281f3b40 con 0x7ff2280c51a0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff22d893640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2280c51a0 0x7ff22815b260 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff218009ac0 tx=0x7ff218004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff21effd640 1 -- 172.21.15.6:0/45652984 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218039050 con 0x7ff2280c51a0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff21effd640 1 -- 172.21.15.6:0/45652984 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff218045070 con 0x7ff2280c51a0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2281f3e20 con 0x7ff2280c51a0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff21effd640 1 -- 172.21.15.6:0/45652984 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2180044e0 con 0x7ff2280c51a0 2024-03-18T14:43:34.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.362+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2281f43e0 con 0x7ff2280c51a0 2024-03-18T14:43:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.363+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2280751b0 con 0x7ff2280c51a0 2024-03-18T14:43:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.364+0000 7ff21effd640 1 -- 172.21.15.6:0/45652984 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff218003850 con 0x7ff2280c51a0 2024-03-18T14:43:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.364+0000 7ff21effd640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff20c06bc00 0x7ff20c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.364+0000 7ff21effd640 1 -- 172.21.15.6:0/45652984 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7ff2180ac3a0 con 0x7ff2280c51a0 2024-03-18T14:43:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.364+0000 7ff22e094640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff20c06bc00 0x7ff20c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.365+0000 7ff22e094640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff20c06bc00 0x7ff20c06e0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff224005530 tx=0x7ff224009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.368+0000 7ff21effd640 1 -- 172.21.15.6:0/45652984 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff21804e020 con 0x7ff2280c51a0 2024-03-18T14:43:34.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.550+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff22815bb80 con 0x7ff2280c51a0 2024-03-18T14:43:34.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.550+0000 7ff21effd640 1 -- 172.21.15.6:0/45652984 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7ff21807b120 con 0x7ff2280c51a0 2024-03-18T14:43:34.552 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff20c06bc00 msgr2=0x7ff20c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff20c06bc00 0x7ff20c06e0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff224005530 tx=0x7ff224009040 comp rx=0 tx=0).stop 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2280c51a0 msgr2=0x7ff22815b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2280c51a0 0x7ff22815b260 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff218009ac0 tx=0x7ff218004730 comp rx=0 tx=0).stop 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 shutdown_connections 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff20c06bc00 0x7ff20c06e0c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22815b7a0 0x7ff2280c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2280c51a0 0x7ff22815b260 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 --2- 172.21.15.6:0/45652984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2140040d0 0x7ff2280c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 >> 172.21.15.6:0/45652984 conn(0x7ff228099a50 msgr2=0x7ff2280701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 shutdown_connections 2024-03-18T14:43:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.552+0000 7ff2302f8640 1 -- 172.21.15.6:0/45652984 wait complete. 2024-03-18T14:43:34.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-18T14:43:34.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.665+0000 7fe3dff59640 1 Processor -- start 2024-03-18T14:43:34.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.665+0000 7fe3dff59640 1 -- start start 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dff59640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3d815bab0 0x7fe3d815be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dff59640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3d806dbb0 0x7fe3d81602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dff59640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806e2b0 0x7fe3d806e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dff59640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe3d8160820 con 0x7fe3d806dbb0 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dff59640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe3d80b2dd0 con 0x7fe3d815bab0 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dff59640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3d80b30b0 con 0x7fe3d806e2b0 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3d806dbb0 0x7fe3d81602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3d806dbb0 0x7fe3d81602e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57298/0 (socket says 172.21.15.6:57298) 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 -- 172.21.15.6:0/2046673576 learned_addr learned my addr 172.21.15.6:0/2046673576 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3ddcf5640 1 --2- 172.21.15.6:0/2046673576 >> v2:172.21.15.92:3300/0 conn(0x7fe3d815bab0 0x7fe3d815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3de4f6640 1 --2- 172.21.15.6:0/2046673576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806e2b0 0x7fe3d806e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 -- 172.21.15.6:0/2046673576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806e2b0 msgr2=0x7fe3d806e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 --2- 172.21.15.6:0/2046673576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806e2b0 0x7fe3d806e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 -- 172.21.15.6:0/2046673576 >> v2:172.21.15.92:3300/0 conn(0x7fe3d815bab0 msgr2=0x7fe3d815be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 --2- 172.21.15.6:0/2046673576 >> v2:172.21.15.92:3300/0 conn(0x7fe3d815bab0 0x7fe3d815be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3dd4f4640 1 -- 172.21.15.6:0/2046673576 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d80b3390 con 0x7fe3d806dbb0 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.666+0000 7fe3de4f6640 1 --2- 172.21.15.6:0/2046673576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806e2b0 0x7fe3d806e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dd4f4640 1 --2- 172.21.15.6:0/2046673576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d806dbb0 0x7fe3d81602e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8072e00 tx=0x7fe3c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecbd791163cda56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dccf3640 1 -- 172.21.15.6:0/2046673576 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8035020 con 0x7fe3d806dbb0 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dccf3640 1 -- 172.21.15.6:0/2046673576 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c8031870 con 0x7fe3d806dbb0 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dccf3640 1 -- 172.21.15.6:0/2046673576 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8031c40 con 0x7fe3d806dbb0 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dff59640 1 -- 172.21.15.6:0/2046673576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d806dbb0 msgr2=0x7fe3d81602e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2046673576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d806dbb0 0x7fe3d81602e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8072e00 tx=0x7fe3c802b910 comp rx=0 tx=0).stop 2024-03-18T14:43:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dff59640 1 -- 172.21.15.6:0/2046673576 shutdown_connections 2024-03-18T14:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2046673576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806e2b0 0x7fe3d806e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2046673576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d806dbb0 0x7fe3d81602e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2046673576 >> v2:172.21.15.92:3300/0 conn(0x7fe3d815bab0 0x7fe3d815be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.667+0000 7fe3dff59640 1 -- 172.21.15.6:0/2046673576 >> 172.21.15.6:0/2046673576 conn(0x7fe3d80b1470 msgr2=0x7fe3d80b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.668+0000 7fe3dff59640 1 -- 172.21.15.6:0/2046673576 shutdown_connections 2024-03-18T14:43:34.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.668+0000 7fe3dff59640 1 -- 172.21.15.6:0/2046673576 wait complete. 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 Processor -- start 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 -- start start 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806dbb0 0x7fe3d80ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3d806e2b0 0x7fe3d80ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d815bab0 0x7fe3d815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d8071ae0 con 0x7fe3d815bab0 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d8071960 con 0x7fe3d806e2b0 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.669+0000 7fe3dff59640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3d8071c60 con 0x7fe3d806dbb0 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d815bab0 0x7fe3d815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d815bab0 0x7fe3d815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57302/0 (socket says 172.21.15.6:57302) 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 -- 172.21.15.6:0/2813912641 learned_addr learned my addr 172.21.15.6:0/2813912641 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3ddcf5640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806dbb0 0x7fe3d80ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3dd4f4640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3d806e2b0 0x7fe3d80ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 -- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806dbb0 msgr2=0x7fe3d80ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806dbb0 0x7fe3d80ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 -- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3d806e2b0 msgr2=0x7fe3d80ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3d806e2b0 0x7fe3d80ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 -- 172.21.15.6:0/2813912641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d80c7c50 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3de4f6640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d815bab0 0x7fe3d815ae40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0007e10 tx=0x7fe3d000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3ce7fc640 1 -- 172.21.15.6:0/2813912641 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d001a070 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3ce7fc640 1 -- 172.21.15.6:0/2813912641 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d0003b90 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3d80ad820 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3ce7fc640 1 -- 172.21.15.6:0/2813912641 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0004030 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.670+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3d80c7ec0 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.671+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3d8073310 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.672+0000 7fe3ce7fc640 1 -- 172.21.15.6:0/2813912641 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe3d001d070 con 0x7fe3d815bab0 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.672+0000 7fe3ce7fc640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3bc06bbd0 0x7fe3bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.672+0000 7fe3ce7fc640 1 -- 172.21.15.6:0/2813912641 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7fe3d008ab50 con 0x7fe3d815bab0 2024-03-18T14:43:34.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.672+0000 7fe3ddcf5640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3bc06bbd0 0x7fe3bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:34.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.673+0000 7fe3ddcf5640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3bc06bbd0 0x7fe3bc06e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4009970 tx=0x7fe3d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:34.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.676+0000 7fe3ce7fc640 1 -- 172.21.15.6:0/2813912641 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe3d80ad820 con 0x7fe3d815bab0 2024-03-18T14:43:34.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:34.928+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7fe3d80c9910 con 0x7fe3d815bab0 2024-03-18T14:43:35.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:43:35.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:43:35.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.928+0000 7fe3ce7fc640 1 -- 172.21.15.6:0/2813912641 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v140) ==== 137+0+0 (secure 0 0 0) 0x7fe3d005f020 con 0x7fe3d815bab0 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3bc06bbd0 msgr2=0x7fe3bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3bc06bbd0 0x7fe3bc06e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4009970 tx=0x7fe3d4008040 comp rx=0 tx=0).stop 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d815bab0 msgr2=0x7fe3d815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d815bab0 0x7fe3d815ae40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0007e10 tx=0x7fe3d000d4a0 comp rx=0 tx=0).stop 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 shutdown_connections 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe3bc06bbd0 0x7fe3bc06e090 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d815bab0 0x7fe3d815ae40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3d806e2b0 0x7fe3d80ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 --2- 172.21.15.6:0/2813912641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3d806dbb0 0x7fe3d80ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 >> 172.21.15.6:0/2813912641 conn(0x7fe3d80b1470 msgr2=0x7fe3d80ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 shutdown_connections 2024-03-18T14:43:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:35.931+0000 7fe3dff59640 1 -- 172.21.15.6:0/2813912641 wait complete. 2024-03-18T14:43:35.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T14:43:36.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.056+0000 7ff57991b640 1 Processor -- start 2024-03-18T14:43:36.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.056+0000 7ff57991b640 1 -- start start 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff57991b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff57406b8d0 0x7ff5740b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff57991b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff5740cc9f0 0x7ff5740cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff57991b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5740b12b0 con 0x7ff5740b0ed0 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff57991b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff5740d18a0 con 0x7ff57406b8d0 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff57991b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff5740d0c90 con 0x7ff5740cc9f0 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5727fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff57406b8d0 0x7ff5740b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5727fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff57406b8d0 0x7ff5740b1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57308/0 (socket says 172.21.15.6:57308) 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5727fc640 1 -- 172.21.15.6:0/1170414974 learned_addr learned my addr 172.21.15.6:0/1170414974 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5737fe640 1 --2- 172.21.15.6:0/1170414974 >> v2:172.21.15.92:3300/0 conn(0x7ff5740cc9f0 0x7ff5740cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5727fc640 1 -- 172.21.15.6:0/1170414974 >> v2:172.21.15.92:3300/0 conn(0x7ff5740cc9f0 msgr2=0x7ff5740cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5727fc640 1 --2- 172.21.15.6:0/1170414974 >> v2:172.21.15.92:3300/0 conn(0x7ff5740cc9f0 0x7ff5740cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5727fc640 1 -- 172.21.15.6:0/1170414974 >> v1:172.21.15.92:6789/0 conn(0x7ff5740b0ed0 legacy=0x7ff57406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5727fc640 1 -- 172.21.15.6:0/1170414974 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5740d0f70 con 0x7ff57406b8d0 2024-03-18T14:43:36.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.057+0000 7ff5737fe640 1 --2- 172.21.15.6:0/1170414974 >> v2:172.21.15.92:3300/0 conn(0x7ff5740cc9f0 0x7ff5740cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:36.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.058+0000 7ff5727fc640 1 --2- 172.21.15.6:0/1170414974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff57406b8d0 0x7ff5740b1e60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff5740cb000 tx=0x7ff56802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18c77eb13fa9dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.058+0000 7ff571ffb640 1 -- 172.21.15.6:0/1170414974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568034070 con 0x7ff57406b8d0 2024-03-18T14:43:36.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.058+0000 7ff571ffb640 1 -- 172.21.15.6:0/1170414974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff568030860 con 0x7ff57406b8d0 2024-03-18T14:43:36.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.058+0000 7ff571ffb640 1 -- 172.21.15.6:0/1170414974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568030c30 con 0x7ff57406b8d0 2024-03-18T14:43:36.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 -- 172.21.15.6:0/1170414974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff57406b8d0 msgr2=0x7ff5740b1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 --2- 172.21.15.6:0/1170414974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff57406b8d0 0x7ff5740b1e60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff5740cb000 tx=0x7ff56802b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 -- 172.21.15.6:0/1170414974 shutdown_connections 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 --2- 172.21.15.6:0/1170414974 >> v2:172.21.15.92:3300/0 conn(0x7ff5740cc9f0 0x7ff5740cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 --2- 172.21.15.6:0/1170414974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff57406b8d0 0x7ff5740b1e60 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 -- 172.21.15.6:0/1170414974 >> 172.21.15.6:0/1170414974 conn(0x7ff574099a50 msgr2=0x7ff574099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 -- 172.21.15.6:0/1170414974 shutdown_connections 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.059+0000 7ff57991b640 1 -- 172.21.15.6:0/1170414974 wait complete. 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.060+0000 7ff57991b640 1 Processor -- start 2024-03-18T14:43:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.060+0000 7ff57991b640 1 -- start start 2024-03-18T14:43:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff57991b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff57406b8d0 0x7ff57415fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff57991b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5740b0ed0 0x7ff57415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff57991b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5740cc9f0 0x7ff5741e7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff57991b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff5740d08f0 con 0x7ff5740cc9f0 2024-03-18T14:43:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff57991b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff5740d1530 con 0x7ff5740b0ed0 2024-03-18T14:43:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff57991b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff5740d0a70 con 0x7ff57406b8d0 2024-03-18T14:43:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5740cc9f0 0x7ff5741e7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5740cc9f0 0x7ff5741e7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57322/0 (socket says 172.21.15.6:57322) 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 -- 172.21.15.6:0/3142576585 learned_addr learned my addr 172.21.15.6:0/3142576585 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff572ffd640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff57406b8d0 0x7ff57415fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5727fc640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5740b0ed0 0x7ff57415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 -- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff57406b8d0 msgr2=0x7ff57415fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff57406b8d0 0x7ff57415fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 -- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5740b0ed0 msgr2=0x7ff57415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5740b0ed0 0x7ff57415ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 -- 172.21.15.6:0/3142576585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff574141570 con 0x7ff5740cc9f0 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff5737fe640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5740cc9f0 0x7ff5741e7be0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff56c007ef0 tx=0x7ff56c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff55bfff640 1 -- 172.21.15.6:0/3142576585 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c01a070 con 0x7ff5740cc9f0 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.061+0000 7ff55bfff640 1 -- 172.21.15.6:0/3142576585 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56c003b90 con 0x7ff5740cc9f0 2024-03-18T14:43:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.062+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57415eb80 con 0x7ff5740cc9f0 2024-03-18T14:43:36.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.062+0000 7ff55bfff640 1 -- 172.21.15.6:0/3142576585 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c004030 con 0x7ff5740cc9f0 2024-03-18T14:43:36.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.062+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57415f420 con 0x7ff5740cc9f0 2024-03-18T14:43:36.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.062+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57404f5b0 con 0x7ff5740cc9f0 2024-03-18T14:43:36.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.063+0000 7ff55bfff640 1 -- 172.21.15.6:0/3142576585 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff56c01d070 con 0x7ff5740cc9f0 2024-03-18T14:43:36.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.063+0000 7ff55bfff640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff56006bbd0 0x7ff56006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.063+0000 7ff55bfff640 1 -- 172.21.15.6:0/3142576585 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7ff56c08aac0 con 0x7ff5740cc9f0 2024-03-18T14:43:36.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.064+0000 7ff572ffd640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff56006bbd0 0x7ff56006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.064+0000 7ff572ffd640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff56006bbd0 0x7ff56006e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff5640016e0 tx=0x7ff564009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.068+0000 7ff55bfff640 1 -- 172.21.15.6:0/3142576585 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff56c010070 con 0x7ff5740cc9f0 2024-03-18T14:43:36.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.250+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff5740b6680 con 0x7ff5740cc9f0 2024-03-18T14:43:36.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.252+0000 7ff55bfff640 1 -- 172.21.15.6:0/3142576585 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7ff56c0590d0 con 0x7ff5740cc9f0 2024-03-18T14:43:36.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff56006bbd0 msgr2=0x7ff56006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff56006bbd0 0x7ff56006e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff5640016e0 tx=0x7ff564009040 comp rx=0 tx=0).stop 2024-03-18T14:43:36.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5740cc9f0 msgr2=0x7ff5741e7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5740cc9f0 0x7ff5741e7be0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff56c007ef0 tx=0x7ff56c00d4a0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 shutdown_connections 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff56006bbd0 0x7ff56006e090 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff5740cc9f0 0x7ff5741e7be0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5740b0ed0 0x7ff57415ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 --2- 172.21.15.6:0/3142576585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff57406b8d0 0x7ff57415fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 >> 172.21.15.6:0/3142576585 conn(0x7ff574099a50 msgr2=0x7ff57406b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 shutdown_connections 2024-03-18T14:43:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.255+0000 7ff57991b640 1 -- 172.21.15.6:0/3142576585 wait complete. 2024-03-18T14:43:36.268 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T14:43:36.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:43:36.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:43:36.362 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T14:43:36.362 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:36.362 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:36.377 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2314 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-18T14:43:36.377 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:43:36.377 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:36.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:43:36.394 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:36.394 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:36.431 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2314 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-18T14:43:36.431 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:43:36.431 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:36.451 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:43:36.451 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T14:43:36.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.570+0000 7f278e6f8640 1 Processor -- start 2024-03-18T14:43:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.570+0000 7f278e6f8640 1 -- start start 2024-03-18T14:43:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f278e6f8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 0x7f27880d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f278e6f8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27880b9660 con 0x7f278815e020 2024-03-18T14:43:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f278e6f8640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2788162810 con 0x7f27880ced00 2024-03-18T14:43:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f278e6f8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f27880d2270 con 0x7f2788162380 2024-03-18T14:43:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f2787fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f278815e020 0x7f27880d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56980/0 (socket says 172.21.15.6:56980) 2024-03-18T14:43:36.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f2787fff640 1 -- 172.21.15.6:0/84440636 learned_addr learned my addr 172.21.15.6:0/84440636 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f278cc95640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 0x7f27880d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1197115680 0 0) 0x7f27880b9660 con 0x7f278815e020 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.571+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2770003520 con 0x7f278815e020 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2260472319 0 0) 0x7f2770003520 con 0x7f278815e020 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 msgr2=0x7f27880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 0x7f27880d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 >> v1:172.21.15.92:6789/0 conn(0x7f27880ced00 legacy=0x7f27880d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27880ce390 con 0x7f278815e020 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f277c001670 con 0x7f278815e020 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 >> v1:172.21.15.6:6789/0 conn(0x7f278815e020 legacy=0x7f27880d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27700039e0 0x7f2770003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 0x7f2770004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27880d2270 con 0x7f2770004d40 2024-03-18T14:43:36.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f277000baa0 con 0x7f27700039e0 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f277000bca0 con 0x7f2788162380 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f278cc95640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f27877fe640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 0x7f2770004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f2787fff640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27700039e0 0x7f2770003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f278cc95640 1 -- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 msgr2=0x7f2770004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f278cc95640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2788162380 0x7f2770004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f278cc95640 1 -- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27700039e0 msgr2=0x7f2770003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f278cc95640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27700039e0 0x7f2770003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.572+0000 7f278cc95640 1 -- 172.21.15.6:0/84440636 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2770003520 con 0x7f2770004d40 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.573+0000 7f278cc95640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27700051c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2780009fc0 tx=0x7f2770005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e0bd4515d4bfcd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.573+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780039070 con 0x7f2770004d40 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.573+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f278002cc20 con 0x7f2770004d40 2024-03-18T14:43:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.573+0000 7f2786ffd640 1 -- 172.21.15.6:0/84440636 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780009c10 con 0x7f2770004d40 2024-03-18T14:43:36.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.574+0000 7f278e6f8640 1 -- 172.21.15.6:0/84440636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 msgr2=0x7f27700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.574+0000 7f278e6f8640 1 --2- 172.21.15.6:0/84440636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27700051c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2780009fc0 tx=0x7f2770005830 comp rx=0 tx=0).stop 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.574+0000 7f278cc95640 1 -- 172.21.15.6:0/84440636 reap_dead start 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.574+0000 7f278e6f8640 1 -- 172.21.15.6:0/84440636 shutdown_connections 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.574+0000 7f278e6f8640 1 -- 172.21.15.6:0/84440636 >> 172.21.15.6:0/84440636 conn(0x7f2788072ce0 msgr2=0x7f27880730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.574+0000 7f278e6f8640 1 -- 172.21.15.6:0/84440636 shutdown_connections 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.574+0000 7f278e6f8640 1 -- 172.21.15.6:0/84440636 wait complete. 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.575+0000 7f278e6f8640 1 Processor -- start 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.575+0000 7f278e6f8640 1 -- start start 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27881f7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f278815fbe0 0x7f278815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27881f3970 0x7f27881f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27880d3920 con 0x7f2770004d40 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f27880b9be0 con 0x7f27881f3970 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f27880ce890 con 0x7f278815fbe0 2024-03-18T14:43:36.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27881f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27881f7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57342/0 (socket says 172.21.15.6:57342) 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 -- 172.21.15.6:0/4098054299 learned_addr learned my addr 172.21.15.6:0/4098054299 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f27877fe640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f278815fbe0 0x7f278815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278cc95640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27881f3970 0x7f27881f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 -- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f278815fbe0 msgr2=0x7f278815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f278815fbe0 0x7f278815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 -- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27881f3970 msgr2=0x7f27881f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27881f3970 0x7f27881f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 -- 172.21.15.6:0/4098054299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27880b37a0 con 0x7f2770004d40 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278cc95640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27881f3970 0x7f27881f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2787fff640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27881f7950 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f277c007290 tx=0x7f277c007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2784ff9640 1 -- 172.21.15.6:0/4098054299 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c00f940 con 0x7f2770004d40 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2784ff9640 1 -- 172.21.15.6:0/4098054299 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277c018070 con 0x7f2770004d40 2024-03-18T14:43:36.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27880b3a80 con 0x7f2770004d40 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f2784ff9640 1 -- 172.21.15.6:0/4098054299 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c00fd40 con 0x7f2770004d40 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.576+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27880b4040 con 0x7f2770004d40 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.577+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278808afe0 con 0x7f2770004d40 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.578+0000 7f2784ff9640 1 -- 172.21.15.6:0/4098054299 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f277c01d460 con 0x7f2770004d40 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.578+0000 7f2784ff9640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f277406bc00 0x7f277406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.578+0000 7f2784ff9640 1 -- 172.21.15.6:0/4098054299 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7f277c08a970 con 0x7f2770004d40 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.578+0000 7f27877fe640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f277406bc00 0x7f277406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.579+0000 7f27877fe640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f277406bc00 0x7f277406e0c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f27881f6ea0 tx=0x7f2778009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.583+0000 7f2784ff9640 1 -- 172.21.15.6:0/4098054299 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f277c0595d0 con 0x7f2770004d40 2024-03-18T14:43:36.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.806+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f278804f620 con 0x7f2770004d40 2024-03-18T14:43:36.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.807+0000 7f2784ff9640 1 -- 172.21.15.6:0/4098054299 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3857 (secure 0 0 0) 0x7f277c058ed0 con 0x7f2770004d40 2024-03-18T14:43:36.809 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:36.809 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/154814120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":154814120},{"type":"v1","addr":"172.21.15.6:6835","nonce":154814120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5141,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/2314779232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2314779232},{"type":"v1","addr":"172.21.15.92:6836","nonce":2314779232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/729844882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":729844882},{"type":"v1","addr":"172.21.15.92:6837","nonce":729844882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":6141,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3554694184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3554694184},{"type":"v1","addr":"172.21.15.6:6834","nonce":3554694184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-03-18T14:43:36.809 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-03-18T14:43:36.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f277406bc00 msgr2=0x7f277406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f277406bc00 0x7f277406e0c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f27881f6ea0 tx=0x7f2778009040 comp rx=0 tx=0).stop 2024-03-18T14:43:36.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 msgr2=0x7f27881f7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27881f7950 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f277c007290 tx=0x7f277c007750 comp rx=0 tx=0).stop 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 shutdown_connections 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f277406bc00 0x7f277406e0c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27881f3970 0x7f27881f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f278815fbe0 0x7f278815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 --2- 172.21.15.6:0/4098054299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2770004d40 0x7f27881f7950 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 >> 172.21.15.6:0/4098054299 conn(0x7f2788072ce0 msgr2=0x7f27881616b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 shutdown_connections 2024-03-18T14:43:36.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.809+0000 7f278e6f8640 1 -- 172.21.15.6:0/4098054299 wait complete. 2024-03-18T14:43:36.822 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T14:43:36.822 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T14:43:36.822 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T14:43:36.823 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:36.823 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T14:43:36.823 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T14:43:36.823 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T14:43:36.823 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:36.823 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T14:43:36.823 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T14:43:36.824 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T14:43:36.824 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:36.824 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T14:43:36.824 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T14:43:36.824 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T14:43:36.824 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:36.832 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T14:43:36.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T14:43:36.832 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T14:43:36.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T14:43:36.836 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T14:43:36.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T14:43:36.836 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T14:43:36.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T14:43:36.874 INFO:tasks.ceph.mds.a:Started 2024-03-18T14:43:36.875 DEBUG:teuthology.parallel:result is None 2024-03-18T14:43:36.875 INFO:tasks.ceph.mds.c:Started 2024-03-18T14:43:36.875 DEBUG:teuthology.parallel:result is None 2024-03-18T14:43:36.878 INFO:tasks.ceph.mds.b:Started 2024-03-18T14:43:36.878 DEBUG:teuthology.parallel:result is None 2024-03-18T14:43:36.879 INFO:tasks.ceph.mds.d:Started 2024-03-18T14:43:36.879 DEBUG:teuthology.parallel:result is None 2024-03-18T14:43:36.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T14:43:36.951 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-18T14:43:36.952 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-18T14:43:36.966 INFO:tasks.ceph.mds.d.smithi092.stdout:starting mds.d at 2024-03-18T14:43:36.966 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2024-03-18T14:43:36.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.984+0000 7f066dcea640 1 Processor -- start 2024-03-18T14:43:36.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.984+0000 7f066dcea640 1 -- start start 2024-03-18T14:43:36.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.984+0000 7f066dcea640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066815dd70 0x7f066815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.984+0000 7f066dcea640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680cecc0 0x7f06680d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.984+0000 7f066dcea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06680d12a0 con 0x7f06680d0910 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.984+0000 7f066dcea640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0668161a50 con 0x7f066815dd70 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.984+0000 7f066dcea640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0668161d30 con 0x7f06680cecc0 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f06677fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066815dd70 0x7f066815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f0667fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f06680d0910 0x7f06680b8eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56998/0 (socket says 172.21.15.6:56998) 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f06677fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066815dd70 0x7f066815ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57390/0 (socket says 172.21.15.6:57390) 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f0667fff640 1 -- 172.21.15.6:0/2973018367 learned_addr learned my addr 172.21.15.6:0/2973018367 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f0666ffd640 1 --2- 172.21.15.6:0/2973018367 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680cecc0 0x7f06680d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f06677fe640 1 -- 172.21.15.6:0/2973018367 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680cecc0 msgr2=0x7f06680d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f06677fe640 1 --2- 172.21.15.6:0/2973018367 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680cecc0 0x7f06680d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f06677fe640 1 -- 172.21.15.6:0/2973018367 >> v1:172.21.15.6:6789/0 conn(0x7f06680d0910 legacy=0x7f06680b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f06677fe640 1 -- 172.21.15.6:0/2973018367 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0668162010 con 0x7f066815dd70 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f06667fc640 1 -- 172.21.15.6:0/2973018367 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 634593415 0 0) 0x7f06680d12a0 con 0x7f06680d0910 2024-03-18T14:43:36.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.985+0000 7f0666ffd640 1 --2- 172.21.15.6:0/2973018367 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680cecc0 0x7f06680d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f06677fe640 1 --2- 172.21.15.6:0/2973018367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066815dd70 0x7f066815ee60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f065c0097a0 tx=0x7f065c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef2e5b21dc2d9a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f06667fc640 1 -- 172.21.15.6:0/2973018367 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c035020 con 0x7f066815dd70 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f06667fc640 1 -- 172.21.15.6:0/2973018367 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f065c031830 con 0x7f066815dd70 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f06667fc640 1 -- 172.21.15.6:0/2973018367 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c031c00 con 0x7f066815dd70 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 -- 172.21.15.6:0/2973018367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066815dd70 msgr2=0x7f066815ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 --2- 172.21.15.6:0/2973018367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066815dd70 0x7f066815ee60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f065c0097a0 tx=0x7f065c02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 -- 172.21.15.6:0/2973018367 shutdown_connections 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 --2- 172.21.15.6:0/2973018367 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680cecc0 0x7f06680d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 --2- 172.21.15.6:0/2973018367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066815dd70 0x7f066815ee60 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 -- 172.21.15.6:0/2973018367 >> 172.21.15.6:0/2973018367 conn(0x7f066806c900 msgr2=0x7f066806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:36.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 -- 172.21.15.6:0/2973018367 shutdown_connections 2024-03-18T14:43:36.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.986+0000 7f066dcea640 1 -- 172.21.15.6:0/2973018367 wait complete. 2024-03-18T14:43:36.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.987+0000 7f066dcea640 1 Processor -- start 2024-03-18T14:43:36.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f066dcea640 1 -- start start 2024-03-18T14:43:36.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f066dcea640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06680cecc0 0x7f06680c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f066dcea640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680d0910 0x7f06680cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f066dcea640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f066815dd70 0x7f06680c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f066dcea640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f066815e150 con 0x7f06680cecc0 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f066dcea640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f06680d0da0 con 0x7f066815dd70 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f066dcea640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f066815e2d0 con 0x7f06680d0910 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f06677fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06680cecc0 0x7f06680c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f06677fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06680cecc0 0x7f06680c70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57396/0 (socket says 172.21.15.6:57396) 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.988+0000 7f06677fe640 1 -- 172.21.15.6:0/3864502678 learned_addr learned my addr 172.21.15.6:0/3864502678 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f0667fff640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f066815dd70 0x7f06680c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f0666ffd640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680d0910 0x7f06680cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f06677fe640 1 -- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680d0910 msgr2=0x7f06680cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f06677fe640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680d0910 0x7f06680cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f06677fe640 1 -- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f066815dd70 msgr2=0x7f06680c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f06677fe640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f066815dd70 0x7f06680c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f06677fe640 1 -- 172.21.15.6:0/3864502678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06680b5540 con 0x7f06680cecc0 2024-03-18T14:43:36.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f06677fe640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06680cecc0 0x7f06680c70e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f065c034040 tx=0x7f065c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f064ffff640 1 -- 172.21.15.6:0/3864502678 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c040070 con 0x7f06680cecc0 2024-03-18T14:43:36.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f064ffff640 1 -- 172.21.15.6:0/3864502678 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f065c0399a0 con 0x7f06680cecc0 2024-03-18T14:43:36.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f064ffff640 1 -- 172.21.15.6:0/3864502678 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c035040 con 0x7f06680cecc0 2024-03-18T14:43:36.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06681f8020 con 0x7f06680cecc0 2024-03-18T14:43:36.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.989+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06680b30d0 con 0x7f06680cecc0 2024-03-18T14:43:36.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.990+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f066804f5b0 con 0x7f06680cecc0 2024-03-18T14:43:36.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.991+0000 7f064ffff640 1 -- 172.21.15.6:0/3864502678 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f065c038020 con 0x7f06680cecc0 2024-03-18T14:43:36.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.991+0000 7f064ffff640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f065406bbd0 0x7f065406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:36.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.991+0000 7f064ffff640 1 -- 172.21.15.6:0/3864502678 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5335+0+0 (secure 0 0 0) 0x7f065c0ac260 con 0x7f06680cecc0 2024-03-18T14:43:36.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.991+0000 7f0666ffd640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f065406bbd0 0x7f065406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:36.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.992+0000 7f0666ffd640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f065406bbd0 0x7f065406e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0658006fd0 tx=0x7f0658008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:36.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:36.995+0000 7f064ffff640 1 -- 172.21.15.6:0/3864502678 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f065c07aed0 con 0x7f06680cecc0 2024-03-18T14:43:37.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.191+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f0668068450 con 0x7f06680cecc0 2024-03-18T14:43:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.947+0000 7f064ffff640 1 -- 172.21.15.6:0/3864502678 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v50) ==== 97+0+0 (secure 0 0 0) 0x7f065c0ac080 con 0x7f06680cecc0 2024-03-18T14:43:37.949 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-18T14:43:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.949+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f065406bbd0 msgr2=0x7f065406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.949+0000 7f066dcea640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f065406bbd0 0x7f065406e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0658006fd0 tx=0x7f0658008040 comp rx=0 tx=0).stop 2024-03-18T14:43:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.949+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06680cecc0 msgr2=0x7f06680c70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.949+0000 7f066dcea640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06680cecc0 0x7f06680c70e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f065c034040 tx=0x7f065c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 shutdown_connections 2024-03-18T14:43:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f065406bbd0 0x7f065406e090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f066815dd70 0x7f06680c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06680d0910 0x7f06680cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 --2- 172.21.15.6:0/3864502678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06680cecc0 0x7f06680c70e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 >> 172.21.15.6:0/3864502678 conn(0x7f066806c900 msgr2=0x7f066815bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 shutdown_connections 2024-03-18T14:43:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:37.950+0000 7f066dcea640 1 -- 172.21.15.6:0/3864502678 wait complete. 2024-03-18T14:43:37.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T14:43:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.070+0000 7f1ad6e62640 1 Processor -- start 2024-03-18T14:43:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.070+0000 7f1ad6e62640 1 -- start start 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.070+0000 7f1ad6e62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad015bab0 0x7f1ad015be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.070+0000 7f1ad6e62640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ad006e9c0 0x7f1ad01602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.070+0000 7f1ad6e62640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ad00b2dd0 con 0x7f1ad006dbb0 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.070+0000 7f1ad6e62640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad0160820 con 0x7f1ad006e9c0 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.070+0000 7f1ad6e62640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad00b30b0 con 0x7f1ad015bab0 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad53ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ad006e9c0 0x7f1ad01602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad53ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ad006e9c0 0x7f1ad01602e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57406/0 (socket says 172.21.15.6:57406) 2024-03-18T14:43:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad53ff640 1 -- 172.21.15.6:0/582228284 learned_addr learned my addr 172.21.15.6:0/582228284 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad4bfe640 1 --2- 172.21.15.6:0/582228284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad015bab0 0x7f1ad015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad53ff640 1 -- 172.21.15.6:0/582228284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad015bab0 msgr2=0x7f1ad015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad53ff640 1 --2- 172.21.15.6:0/582228284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad015bab0 0x7f1ad015be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad53ff640 1 -- 172.21.15.6:0/582228284 >> v1:172.21.15.92:6789/0 conn(0x7f1ad006dbb0 legacy=0x7f1ad006e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.071+0000 7f1ad53ff640 1 -- 172.21.15.6:0/582228284 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad00b3390 con 0x7f1ad006e9c0 2024-03-18T14:43:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad53ff640 1 --2- 172.21.15.6:0/582228284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad006e9c0 0x7f1ad01602e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009e80 tx=0x7f1ac802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9abaa6403d3388c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1acf7fe640 1 -- 172.21.15.6:0/582228284 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac802caf0 con 0x7f1ad006e9c0 2024-03-18T14:43:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1acf7fe640 1 -- 172.21.15.6:0/582228284 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac802cc90 con 0x7f1ad006e9c0 2024-03-18T14:43:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1acf7fe640 1 -- 172.21.15.6:0/582228284 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8034610 con 0x7f1ad006e9c0 2024-03-18T14:43:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 -- 172.21.15.6:0/582228284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad006e9c0 msgr2=0x7f1ad01602e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/582228284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad006e9c0 0x7f1ad01602e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009e80 tx=0x7f1ac802bf50 comp rx=0 tx=0).stop 2024-03-18T14:43:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 -- 172.21.15.6:0/582228284 shutdown_connections 2024-03-18T14:43:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/582228284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad006e9c0 0x7f1ad01602e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/582228284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad015bab0 0x7f1ad015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 -- 172.21.15.6:0/582228284 >> 172.21.15.6:0/582228284 conn(0x7f1ad00b1470 msgr2=0x7f1ad00b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 -- 172.21.15.6:0/582228284 shutdown_connections 2024-03-18T14:43:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.072+0000 7f1ad6e62640 1 -- 172.21.15.6:0/582228284 wait complete. 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.074+0000 7f1ad6e62640 1 Processor -- start 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.074+0000 7f1ad6e62640 1 -- start start 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.074+0000 7f1ad6e62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad006dbb0 0x7f1ad00c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.074+0000 7f1ad6e62640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ad006e9c0 0x7f1ad00c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad6e62640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad015bab0 0x7f1ad00ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad6e62640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad00b3d10 con 0x7f1ad015bab0 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad6e62640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad00b3b90 con 0x7f1ad006e9c0 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad6e62640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad00b3e90 con 0x7f1ad006dbb0 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad015bab0 0x7f1ad00ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad015bab0 0x7f1ad00ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57420/0 (socket says 172.21.15.6:57420) 2024-03-18T14:43:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 -- 172.21.15.6:0/1217254058 learned_addr learned my addr 172.21.15.6:0/1217254058 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1acffff640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ad006e9c0 0x7f1ad00c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad4bfe640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad006dbb0 0x7f1ad00c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 -- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad006dbb0 msgr2=0x7f1ad00c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad006dbb0 0x7f1ad00c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 -- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ad006e9c0 msgr2=0x7f1ad00c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ad006e9c0 0x7f1ad00c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 -- 172.21.15.6:0/1217254058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad00ac970 con 0x7f1ad015bab0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad53ff640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad015bab0 0x7f1ad00ad9a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036040 tx=0x7f1ac8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1acd7fa640 1 -- 172.21.15.6:0/1217254058 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8043020 con 0x7f1ad015bab0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1acd7fa640 1 -- 172.21.15.6:0/1217254058 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac803e450 con 0x7f1ad015bab0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad00acbe0 con 0x7f1ad015bab0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1acd7fa640 1 -- 172.21.15.6:0/1217254058 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8009510 con 0x7f1ad015bab0 2024-03-18T14:43:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.075+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad00ad1a0 con 0x7f1ad015bab0 2024-03-18T14:43:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.077+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad0073310 con 0x7f1ad015bab0 2024-03-18T14:43:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.077+0000 7f1acd7fa640 1 -- 172.21.15.6:0/1217254058 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1ac8046020 con 0x7f1ad015bab0 2024-03-18T14:43:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.078+0000 7f1acd7fa640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1abc06bbd0 0x7f1abc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.078+0000 7f1acd7fa640 1 -- 172.21.15.6:0/1217254058 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7f1ac80acfe0 con 0x7f1ad015bab0 2024-03-18T14:43:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.078+0000 7f1ad4bfe640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1abc06bbd0 0x7f1abc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.079+0000 7f1ad4bfe640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1abc06bbd0 0x7f1abc06e090 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006fd0 tx=0x7f1ac0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.082+0000 7f1acd7fa640 1 -- 172.21.15.6:0/1217254058 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1ac804d030 con 0x7f1ad015bab0 2024-03-18T14:43:38.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.332+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1ad00c3190 con 0x7f1ad015bab0 2024-03-18T14:43:38.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.333+0000 7f1acd7fa640 1 -- 172.21.15.6:0/1217254058 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5298 (secure 0 0 0) 0x7f1ac807bd20 con 0x7f1ad015bab0 2024-03-18T14:43:38.335 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:38.335 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:38.335 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-18T14:43:38.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhltJMFFxAAsrfhgR5dgZaElpPSyFsH7Q==", 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlQARXIxAAcOlIn0OSzieJz4X0FMie1A==", 2024-03-18T14:43:38.336 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhla1pOHhAAsDJ0P/0XWnoBakiPfEPYyA==", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlADqzKhAAfG1fJKDAsis1WSLEAMQq0Q==", 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBpUvhlzhYhDxAATnOtwIggkZq8Cix5xsKd7A==", 2024-03-18T14:43:38.338 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBsUvhlkHrEDRAAQHbA4maLjVfgH5XiBUVn0g==", 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBvUvhlN5f9CxAAor/uSMxdAdq8E9k/Bcay0g==", 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQByUvhls/oKChAANwFAbqxQRefnUxXp74nxaA==", 2024-03-18T14:43:38.340 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQB4Uvhlj6qgEhAA5pcxWwon2a8wJcS4myHh/g==", 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQB7Uvhlx9p2ERAAP20961lxyrshPFvUDivuhA==", 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQB+Uvhl7I1SEBAAPkJjmK4erqFLRfqnbFkFqw==", 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCBUvhldq3UDhAAgwbv2PzFpLHI4YE/Gpg/Eg==", 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q==", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw pool=cephfs_data" 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhl1tD9NxAA7VTlgLnBw8A2KI2Rro0eAQ==", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-18T14:43:38.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlwRCfNBAAvhl6UTpZTW7GDuq1BP+hqQ==", 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-18T14:43:38.345 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBmUvhlByUyAhAANpMa65PITql7fLcswh+vog==", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBkUvhlkIZbNRAAe4cTwo1Xf7UpmD6PIgTc0A==", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-18T14:43:38.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlM+SZExAAawGogzVd1g2EvnpMmL0r1A==", 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQBlUvhlGVnoDhAAblQd6YniwXUXlv3N2eZvKQ==", 2024-03-18T14:43:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-18T14:43:38.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-18T14:43:38.348 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-18T14:43:38.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-18T14:43:38.348 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.348 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:38.348 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-18T14:43:38.348 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:38.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1abc06bbd0 msgr2=0x7f1abc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1abc06bbd0 0x7f1abc06e090 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006fd0 tx=0x7f1ac0008040 comp rx=0 tx=0).stop 2024-03-18T14:43:38.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad015bab0 msgr2=0x7f1ad00ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad015bab0 0x7f1ad00ad9a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036040 tx=0x7f1ac8004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:38.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 shutdown_connections 2024-03-18T14:43:38.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1abc06bbd0 0x7f1abc06e090 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ad015bab0 0x7f1ad00ad9a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ad006e9c0 0x7f1ad00c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 --2- 172.21.15.6:0/1217254058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ad006dbb0 0x7f1ad00c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 >> 172.21.15.6:0/1217254058 conn(0x7f1ad00b1470 msgr2=0x7f1ad00ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:38.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 shutdown_connections 2024-03-18T14:43:38.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.335+0000 7f1ad6e62640 1 -- 172.21.15.6:0/1217254058 wait complete. 2024-03-18T14:43:38.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:38.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.449+0000 7f11a41c6640 1 Processor -- start 2024-03-18T14:43:38.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a41c6640 1 -- start start 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a41c6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f119c0ced00 0x7f119c162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a41c6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c0b9660 con 0x7f119c15e020 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a41c6640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119c0ce390 con 0x7f119c0d3210 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a41c6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f119c0d2270 con 0x7f119c0ced00 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a1f62640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f119c15e020 0x7f119c0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57002/0 (socket says 172.21.15.6:57002) 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a1f62640 1 -- 172.21.15.6:0/1313240796 learned_addr learned my addr 172.21.15.6:0/1313240796 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.450+0000 7f11a1761640 1 --2- 172.21.15.6:0/1313240796 >> v2:172.21.15.6:3300/0 conn(0x7f119c0ced00 0x7f119c162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.451+0000 7f11a1761640 1 -- 172.21.15.6:0/1313240796 >> v1:172.21.15.92:6789/0 conn(0x7f119c0d3210 legacy=0x7f119c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.451+0000 7f11a1761640 1 -- 172.21.15.6:0/1313240796 >> v1:172.21.15.6:6789/0 conn(0x7f119c15e020 legacy=0x7f119c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.451+0000 7f11a0f60640 1 -- 172.21.15.6:0/1313240796 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886229287 0 0) 0x7f119c0b9660 con 0x7f119c15e020 2024-03-18T14:43:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.451+0000 7f11a1761640 1 -- 172.21.15.6:0/1313240796 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c0b9900 con 0x7f119c0ced00 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.451+0000 7f11a1761640 1 --2- 172.21.15.6:0/1313240796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c0ced00 0x7f119c162380 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f11900097a0 tx=0x7f119002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b76d061bcfbc6bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a0f60640 1 -- 172.21.15.6:0/1313240796 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11900315a0 con 0x7f119c0ced00 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a0f60640 1 -- 172.21.15.6:0/1313240796 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1190037070 con 0x7f119c0ced00 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a0f60640 1 -- 172.21.15.6:0/1313240796 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1190031960 con 0x7f119c0ced00 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a41c6640 1 -- 172.21.15.6:0/1313240796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c0ced00 msgr2=0x7f119c162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a41c6640 1 --2- 172.21.15.6:0/1313240796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c0ced00 0x7f119c162380 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f11900097a0 tx=0x7f119002b840 comp rx=0 tx=0).stop 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a41c6640 1 -- 172.21.15.6:0/1313240796 shutdown_connections 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a41c6640 1 --2- 172.21.15.6:0/1313240796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c0ced00 0x7f119c162380 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a41c6640 1 -- 172.21.15.6:0/1313240796 >> 172.21.15.6:0/1313240796 conn(0x7f119c072ce0 msgr2=0x7f119c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a41c6640 1 -- 172.21.15.6:0/1313240796 shutdown_connections 2024-03-18T14:43:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.452+0000 7f11a41c6640 1 -- 172.21.15.6:0/1313240796 wait complete. 2024-03-18T14:43:38.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.453+0000 7f11a41c6640 1 Processor -- start 2024-03-18T14:43:38.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.453+0000 7f11a41c6640 1 -- start start 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a41c6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f119c0ced00 0x7f119c1f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a41c6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c0d3210 0x7f119c15fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a41c6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c15e020 0x7f119c0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a41c6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f119c0cf2c0 con 0x7f119c15e020 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a41c6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f119c0ce890 con 0x7f119c0d3210 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a41c6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f119c0cf440 con 0x7f119c0ced00 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a2763640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c15e020 0x7f119c0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a2763640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c15e020 0x7f119c0b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57438/0 (socket says 172.21.15.6:57438) 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a2763640 1 -- 172.21.15.6:0/3107756893 learned_addr learned my addr 172.21.15.6:0/3107756893 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a1f62640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f119c0ced00 0x7f119c1f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.454+0000 7f11a1761640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c0d3210 0x7f119c15fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a2763640 1 -- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f119c0ced00 msgr2=0x7f119c1f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a2763640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f119c0ced00 0x7f119c1f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a2763640 1 -- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c0d3210 msgr2=0x7f119c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a2763640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c0d3210 0x7f119c15fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a2763640 1 -- 172.21.15.6:0/3107756893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c0b2510 con 0x7f119c15e020 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a1761640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c0d3210 0x7f119c15fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a2763640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c15e020 0x7f119c0b1fd0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f118c00c780 tx=0x7f118c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f119a7fc640 1 -- 172.21.15.6:0/3107756893 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c017070 con 0x7f119c15e020 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f119a7fc640 1 -- 172.21.15.6:0/3107756893 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f118c003da0 con 0x7f119c15e020 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c1f8370 con 0x7f119c15e020 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f119a7fc640 1 -- 172.21.15.6:0/3107756893 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c0052b0 con 0x7f119c15e020 2024-03-18T14:43:38.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.455+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c1f8930 con 0x7f119c15e020 2024-03-18T14:43:38.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.456+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119c08afe0 con 0x7f119c15e020 2024-03-18T14:43:38.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.456+0000 7f119a7fc640 1 -- 172.21.15.6:0/3107756893 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f118c01a070 con 0x7f119c15e020 2024-03-18T14:43:38.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.457+0000 7f119a7fc640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f118806bbd0 0x7f118806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.457+0000 7f119a7fc640 1 -- 172.21.15.6:0/3107756893 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7f118c089eb0 con 0x7f119c15e020 2024-03-18T14:43:38.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.457+0000 7f11a1f62640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f118806bbd0 0x7f118806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.457+0000 7f11a1f62640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f118806bbd0 0x7f118806e090 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1194000f60 tx=0x7f1194009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.461+0000 7f119a7fc640 1 -- 172.21.15.6:0/3107756893 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f118c058bf0 con 0x7f119c15e020 2024-03-18T14:43:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.643+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f119c15d7d0 con 0x7f119c15e020 2024-03-18T14:43:38.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.644+0000 7f119a7fc640 1 -- 172.21.15.6:0/3107756893 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+13068 (secure 0 0 0) 0x7f118c0584f0 con 0x7f119c15e020 2024-03-18T14:43:38.646 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:38.646 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":52,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:37.962125+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f118806bbd0 msgr2=0x7f118806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f118806bbd0 0x7f118806e090 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1194000f60 tx=0x7f1194009040 comp rx=0 tx=0).stop 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c15e020 msgr2=0x7f119c0b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c15e020 0x7f119c0b1fd0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f118c00c780 tx=0x7f118c00cc40 comp rx=0 tx=0).stop 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 shutdown_connections 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f118806bbd0 0x7f118806e090 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f119c15e020 0x7f119c0b1fd0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c0d3210 0x7f119c15fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 --2- 172.21.15.6:0/3107756893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f119c0ced00 0x7f119c1f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.645+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 >> 172.21.15.6:0/3107756893 conn(0x7f119c072ce0 msgr2=0x7f119c0b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.646+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 shutdown_connections 2024-03-18T14:43:38.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.646+0000 7f11a41c6640 1 -- 172.21.15.6:0/3107756893 wait complete. 2024-03-18T14:43:38.659 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:38.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:38.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.761+0000 7f8b8c59d640 1 Processor -- start 2024-03-18T14:43:38.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.761+0000 7f8b8c59d640 1 -- start start 2024-03-18T14:43:38.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b8c59d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8406cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b8c59d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b8406d020 0x7f8b840d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b8c59d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b840d0f30 con 0x7f8b840d0680 2024-03-18T14:43:38.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b8c59d640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8b8406d480 con 0x7f8b8406d020 2024-03-18T14:43:38.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b8c59d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8b840b1070 con 0x7f8b840ce930 2024-03-18T14:43:38.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b89b38640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b8406d020 0x7f8b840d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b89b38640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b8406d020 0x7f8b840d0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57454/0 (socket says 172.21.15.6:57454) 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b8ab3a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8b840d0680 0x7f8b8415d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57008/0 (socket says 172.21.15.6:57008) 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.762+0000 7f8b89b38640 1 -- 172.21.15.6:0/269643767 learned_addr learned my addr 172.21.15.6:0/269643767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.763+0000 7f8b8a339640 1 --2- 172.21.15.6:0/269643767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8406cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.763+0000 7f8b89b38640 1 -- 172.21.15.6:0/269643767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 msgr2=0x7f8b8406cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.763+0000 7f8b89b38640 1 --2- 172.21.15.6:0/269643767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8406cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.763+0000 7f8b89b38640 1 -- 172.21.15.6:0/269643767 >> v1:172.21.15.6:6789/0 conn(0x7f8b840d0680 legacy=0x7f8b8415d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.763+0000 7f8b89b38640 1 -- 172.21.15.6:0/269643767 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b840b1350 con 0x7f8b8406d020 2024-03-18T14:43:38.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.763+0000 7f8b89337640 1 -- 172.21.15.6:0/269643767 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 215194836 0 0) 0x7f8b840d0f30 con 0x7f8b840d0680 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.763+0000 7f8b89b38640 1 --2- 172.21.15.6:0/269643767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b840d0140 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009870 tx=0x7f8b7402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330dc7bf4f337386 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b89337640 1 -- 172.21.15.6:0/269643767 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74034070 con 0x7f8b8406d020 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b89337640 1 -- 172.21.15.6:0/269643767 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b74030840 con 0x7f8b8406d020 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b89337640 1 -- 172.21.15.6:0/269643767 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74030c10 con 0x7f8b8406d020 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 -- 172.21.15.6:0/269643767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 msgr2=0x7f8b840d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/269643767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b840d0140 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009870 tx=0x7f8b7402b980 comp rx=0 tx=0).stop 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 -- 172.21.15.6:0/269643767 shutdown_connections 2024-03-18T14:43:38.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/269643767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b840d0140 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/269643767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8406cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 -- 172.21.15.6:0/269643767 >> 172.21.15.6:0/269643767 conn(0x7f8b8408b0c0 msgr2=0x7f8b840d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:38.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 -- 172.21.15.6:0/269643767 shutdown_connections 2024-03-18T14:43:38.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.764+0000 7f8b8c59d640 1 -- 172.21.15.6:0/269643767 wait complete. 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.765+0000 7f8b8c59d640 1 Processor -- start 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.765+0000 7f8b8c59d640 1 -- start start 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8c59d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b841f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8c59d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8414ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8c59d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b840d0680 0x7f8b840c5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8c59d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8b840cee90 con 0x7f8b8406d020 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8c59d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b840ced10 con 0x7f8b840d0680 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8c59d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8415ddc0 con 0x7f8b840ce930 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8a339640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b841f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8a339640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b841f7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57464/0 (socket says 172.21.15.6:57464) 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8a339640 1 -- 172.21.15.6:0/3621327768 learned_addr learned my addr 172.21.15.6:0/3621327768 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8a339640 1 -- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 msgr2=0x7f8b8414ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b89b38640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8414ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8ab3a640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b840d0680 0x7f8b840c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.766+0000 7f8b8a339640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8414ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b8a339640 1 -- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b840d0680 msgr2=0x7f8b840c5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b8a339640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b840d0680 0x7f8b840c5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b8a339640 1 -- 172.21.15.6:0/3621327768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b841f8460 con 0x7f8b8406d020 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b89b38640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8414ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b8ab3a640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b840d0680 0x7f8b840c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b8a339640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b841f7f20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c820 tx=0x7f8b8000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b7affd640 1 -- 172.21.15.6:0/3621327768 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80015070 con 0x7f8b8406d020 2024-03-18T14:43:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b7affd640 1 -- 172.21.15.6:0/3621327768 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b80007a30 con 0x7f8b8406d020 2024-03-18T14:43:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b840b6900 con 0x7f8b8406d020 2024-03-18T14:43:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b7affd640 1 -- 172.21.15.6:0/3621327768 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80007de0 con 0x7f8b8406d020 2024-03-18T14:43:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.767+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b840b6ec0 con 0x7f8b8406d020 2024-03-18T14:43:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.768+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b840751b0 con 0x7f8b8406d020 2024-03-18T14:43:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.768+0000 7f8b7affd640 1 -- 172.21.15.6:0/3621327768 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8b80018070 con 0x7f8b8406d020 2024-03-18T14:43:38.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.769+0000 7f8b7affd640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8b6806bbd0 0x7f8b6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:38.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.769+0000 7f8b7affd640 1 -- 172.21.15.6:0/3621327768 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7f8b8008a730 con 0x7f8b8406d020 2024-03-18T14:43:38.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.769+0000 7f8b89b38640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8b6806bbd0 0x7f8b6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:38.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.769+0000 7f8b89b38640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8b6806bbd0 0x7f8b6806e090 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8b74036000 tx=0x7f8b7403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:38.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.773+0000 7f8b7affd640 1 -- 172.21.15.6:0/3621327768 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8b841f8460 con 0x7f8b8406d020 2024-03-18T14:43:38.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.955+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8b8408b6b0 con 0x7f8b8406d020 2024-03-18T14:43:38.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.955+0000 7f8b7affd640 1 -- 172.21.15.6:0/3621327768 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7f8b80059470 con 0x7f8b8406d020 2024-03-18T14:43:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8b6806bbd0 msgr2=0x7f8b6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8b6806bbd0 0x7f8b6806e090 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8b74036000 tx=0x7f8b7403b040 comp rx=0 tx=0).stop 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 msgr2=0x7f8b841f7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b841f7f20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c820 tx=0x7f8b8000cce0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 shutdown_connections 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8b6806bbd0 0x7f8b6806e090 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b840d0680 0x7f8b840c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b840ce930 0x7f8b8414ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 --2- 172.21.15.6:0/3621327768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b8406d020 0x7f8b841f7f20 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 >> 172.21.15.6:0/3621327768 conn(0x7f8b8408b0c0 msgr2=0x7f8b840ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 shutdown_connections 2024-03-18T14:43:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:38.957+0000 7f8b8c59d640 1 -- 172.21.15.6:0/3621327768 wait complete. 2024-03-18T14:43:38.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T14:43:39.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.069+0000 7fdac8aa0640 1 Processor -- start 2024-03-18T14:43:39.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.069+0000 7fdac8aa0640 1 -- start start 2024-03-18T14:43:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac8aa0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdac40cdde0 0x7fdac406b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac8aa0640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac406b790 0x7fdac40caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac8aa0640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac40ccd40 con 0x7fdac40b1740 2024-03-18T14:43:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac8aa0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fdac40ce250 con 0x7fdac40cdde0 2024-03-18T14:43:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac8aa0640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdac40cd020 con 0x7fdac406b790 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac2d9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac406b790 0x7fdac40caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdab9d9b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdac40cdde0 0x7fdac406b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac2d9d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac406b790 0x7fdac40caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51464/0 (socket says 172.21.15.6:51464) 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac2d9d640 1 -- 172.21.15.6:0/4094121052 learned_addr learned my addr 172.21.15.6:0/4094121052 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac2d9d640 1 -- 172.21.15.6:0/4094121052 >> v2:172.21.15.92:3300/0 conn(0x7fdac40cdde0 msgr2=0x7fdac406b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac2d9d640 1 --2- 172.21.15.6:0/4094121052 >> v2:172.21.15.92:3300/0 conn(0x7fdac40cdde0 0x7fdac406b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac2d9d640 1 -- 172.21.15.6:0/4094121052 >> v1:172.21.15.92:6789/0 conn(0x7fdac40b1740 legacy=0x7fdac40b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdac2d9d640 1 -- 172.21.15.6:0/4094121052 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac40cd300 con 0x7fdac406b790 2024-03-18T14:43:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.070+0000 7fdab9d9b640 1 --2- 172.21.15.6:0/4094121052 >> v2:172.21.15.92:3300/0 conn(0x7fdac40cdde0 0x7fdac406b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.071+0000 7fdac2d9d640 1 --2- 172.21.15.6:0/4094121052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac406b790 0x7fdac40caff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009e50 tx=0x7fdabc02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=839295c60b05efbd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.071+0000 7fdac1d9b640 1 -- 172.21.15.6:0/4094121052 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc02cb90 con 0x7fdac406b790 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.071+0000 7fdac1d9b640 1 -- 172.21.15.6:0/4094121052 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdabc02cd30 con 0x7fdac406b790 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.071+0000 7fdac1d9b640 1 -- 172.21.15.6:0/4094121052 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc034610 con 0x7fdac406b790 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 -- 172.21.15.6:0/4094121052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac406b790 msgr2=0x7fdac40caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/4094121052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac406b790 0x7fdac40caff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009e50 tx=0x7fdabc02bff0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 -- 172.21.15.6:0/4094121052 shutdown_connections 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/4094121052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac406b790 0x7fdac40caff0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/4094121052 >> v2:172.21.15.92:3300/0 conn(0x7fdac40cdde0 0x7fdac406b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 -- 172.21.15.6:0/4094121052 >> 172.21.15.6:0/4094121052 conn(0x7fdac40d09a0 msgr2=0x7fdac40d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 -- 172.21.15.6:0/4094121052 shutdown_connections 2024-03-18T14:43:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.072+0000 7fdac8aa0640 1 -- 172.21.15.6:0/4094121052 wait complete. 2024-03-18T14:43:39.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.073+0000 7fdac8aa0640 1 Processor -- start 2024-03-18T14:43:39.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac8aa0640 1 -- start start 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac8aa0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac406b790 0x7fdac41d9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac8aa0640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac40b1740 0x7fdac41e2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac8aa0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdac40cdde0 0x7fdac41e2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac8aa0640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdac406bbd0 con 0x7fdac40cdde0 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac8aa0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdac40cd8c0 con 0x7fdac406b790 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac8aa0640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fdac406bd50 con 0x7fdac40b1740 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac2d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdac40cdde0 0x7fdac41e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac2d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdac40cdde0 0x7fdac41e2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57468/0 (socket says 172.21.15.6:57468) 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.074+0000 7fdac2d9d640 1 -- 172.21.15.6:0/1947214260 learned_addr learned my addr 172.21.15.6:0/1947214260 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac259c640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac406b790 0x7fdac41d9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdab9d9b640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac40b1740 0x7fdac41e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac2d9d640 1 -- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac40b1740 msgr2=0x7fdac41e2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac2d9d640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac40b1740 0x7fdac41e2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac2d9d640 1 -- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac406b790 msgr2=0x7fdac41d9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac2d9d640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac406b790 0x7fdac41d9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac2d9d640 1 -- 172.21.15.6:0/1947214260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac41da140 con 0x7fdac40cdde0 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac2d9d640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdac40cdde0 0x7fdac41e2670 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009f70 tx=0x7fdabc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdabb7fe640 1 -- 172.21.15.6:0/1947214260 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc043020 con 0x7fdac40cdde0 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdabb7fe640 1 -- 172.21.15.6:0/1947214260 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdabc03e450 con 0x7fdac40cdde0 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac40b35c0 con 0x7fdac40cdde0 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdabb7fe640 1 -- 172.21.15.6:0/1947214260 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc009510 con 0x7fdac40cdde0 2024-03-18T14:43:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.075+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac40c6f20 con 0x7fdac40cdde0 2024-03-18T14:43:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.076+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac40b2270 con 0x7fdac40cdde0 2024-03-18T14:43:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.077+0000 7fdabb7fe640 1 -- 172.21.15.6:0/1947214260 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdabc046020 con 0x7fdac40cdde0 2024-03-18T14:43:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.077+0000 7fdabb7fe640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaa406bbd0 0x7fdaa406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.077+0000 7fdabb7fe640 1 -- 172.21.15.6:0/1947214260 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7fdabc0ace20 con 0x7fdac40cdde0 2024-03-18T14:43:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.077+0000 7fdac259c640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaa406bbd0 0x7fdaa406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.078+0000 7fdac259c640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaa406bbd0 0x7fdaa406e090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdab4001780 tx=0x7fdab4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:39.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.081+0000 7fdabb7fe640 1 -- 172.21.15.6:0/1947214260 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdabc04d030 con 0x7fdac40cdde0 2024-03-18T14:43:39.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.266+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdac404f620 con 0x7fdac40cdde0 2024-03-18T14:43:39.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.266+0000 7fdabb7fe640 1 -- 172.21.15.6:0/1947214260 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) ==== 84+0+65 (secure 0 0 0) 0x7fdac41da140 con 0x7fdac40cdde0 2024-03-18T14:43:39.269 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:39.269 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-18T14:43:39.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-18T14:43:39.269 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-18T14:43:39.269 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-18T14:43:39.269 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:39.269 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaa406bbd0 msgr2=0x7fdaa406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaa406bbd0 0x7fdaa406e090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdab4001780 tx=0x7fdab4009040 comp rx=0 tx=0).stop 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdac40cdde0 msgr2=0x7fdac41e2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdac40cdde0 0x7fdac41e2670 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009f70 tx=0x7fdabc004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 shutdown_connections 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fdaa406bbd0 0x7fdaa406e090 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdac40cdde0 0x7fdac41e2670 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdac40b1740 0x7fdac41e2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 --2- 172.21.15.6:0/1947214260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdac406b790 0x7fdac41d9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 >> 172.21.15.6:0/1947214260 conn(0x7fdac40d09a0 msgr2=0x7fdac40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 shutdown_connections 2024-03-18T14:43:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.268+0000 7fdac8aa0640 1 -- 172.21.15.6:0/1947214260 wait complete. 2024-03-18T14:43:39.281 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T14:43:39.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T14:43:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.384+0000 7f2e0b73b640 1 Processor -- start 2024-03-18T14:43:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.384+0000 7f2e0b73b640 1 -- start start 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e0b73b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e04159a60 0x7f2e0406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e0b73b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e040c4080 0x7f2e0406b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e0b73b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e040b1a50 con 0x7f2e0415cc50 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e0b73b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f2e040b1cf0 con 0x7f2e040c4080 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e0b73b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2e040b1fd0 con 0x7f2e04159a60 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e08cd6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e040c4080 0x7f2e0406b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e09cd8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2e0415cc50 0x7f2e0415f820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57010/0 (socket says 172.21.15.6:57010) 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e08cd6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e040c4080 0x7f2e0406b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57480/0 (socket says 172.21.15.6:57480) 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e09cd8640 1 -- 172.21.15.6:0/4253775373 learned_addr learned my addr 172.21.15.6:0/4253775373 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e094d7640 1 --2- 172.21.15.6:0/4253775373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e04159a60 0x7f2e0406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e08cd6640 1 -- 172.21.15.6:0/4253775373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e04159a60 msgr2=0x7f2e0406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.385+0000 7f2e08cd6640 1 --2- 172.21.15.6:0/4253775373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e04159a60 0x7f2e0406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.386+0000 7f2e08cd6640 1 -- 172.21.15.6:0/4253775373 >> v1:172.21.15.6:6789/0 conn(0x7f2e0415cc50 legacy=0x7f2e0415f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.386+0000 7f2e08cd6640 1 -- 172.21.15.6:0/4253775373 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0415d170 con 0x7f2e040c4080 2024-03-18T14:43:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.386+0000 7f2e03fff640 1 -- 172.21.15.6:0/4253775373 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2790155129 0 0) 0x7f2e040b1a50 con 0x7f2e0415cc50 2024-03-18T14:43:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.386+0000 7f2e08cd6640 1 --2- 172.21.15.6:0/4253775373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040c4080 0x7f2e0406b700 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2df80097a0 tx=0x7f2df802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73994e125d45c7ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.386+0000 7f2e03fff640 1 -- 172.21.15.6:0/4253775373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df80315a0 con 0x7f2e040c4080 2024-03-18T14:43:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.386+0000 7f2e03fff640 1 -- 172.21.15.6:0/4253775373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2df8037070 con 0x7f2e040c4080 2024-03-18T14:43:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.386+0000 7f2e03fff640 1 -- 172.21.15.6:0/4253775373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df8031960 con 0x7f2e040c4080 2024-03-18T14:43:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 -- 172.21.15.6:0/4253775373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040c4080 msgr2=0x7f2e0406b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/4253775373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040c4080 0x7f2e0406b700 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2df80097a0 tx=0x7f2df802b840 comp rx=0 tx=0).stop 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 -- 172.21.15.6:0/4253775373 shutdown_connections 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/4253775373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e040c4080 0x7f2e0406b700 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/4253775373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e04159a60 0x7f2e0406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 -- 172.21.15.6:0/4253775373 >> 172.21.15.6:0/4253775373 conn(0x7f2e040d0980 msgr2=0x7f2e040d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 -- 172.21.15.6:0/4253775373 shutdown_connections 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.387+0000 7f2e0b73b640 1 -- 172.21.15.6:0/4253775373 wait complete. 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.388+0000 7f2e0b73b640 1 Processor -- start 2024-03-18T14:43:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.388+0000 7f2e0b73b640 1 -- start start 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e0b73b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e040c4080 0x7f2e040ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e0b73b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e0415cc50 0x7f2e040af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e0b73b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e04158a40 0x7f2e0414aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e0b73b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0406bc40 con 0x7f2e04158a40 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e0b73b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2e040cddd0 con 0x7f2e040c4080 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e0b73b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f2e0406bdc0 con 0x7f2e0415cc50 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e04158a40 0x7f2e0414aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e04158a40 0x7f2e0414aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57484/0 (socket says 172.21.15.6:57484) 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 -- 172.21.15.6:0/464114323 learned_addr learned my addr 172.21.15.6:0/464114323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e094d7640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e040c4080 0x7f2e040ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e08cd6640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e0415cc50 0x7f2e040af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 -- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e0415cc50 msgr2=0x7f2e040af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e0415cc50 0x7f2e040af4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 -- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e040c4080 msgr2=0x7f2e040ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e040c4080 0x7f2e040ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 -- 172.21.15.6:0/464114323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e040c7d90 con 0x7f2e04158a40 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e094d7640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e040c4080 0x7f2e040ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e09cd8640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e04158a40 0x7f2e0414aa00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c3e0 tx=0x7f2dfc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.389+0000 7f2e01ffb640 1 -- 172.21.15.6:0/464114323 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0074e0 con 0x7f2e04158a40 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.390+0000 7f2e01ffb640 1 -- 172.21.15.6:0/464114323 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dfc0047e0 con 0x7f2e04158a40 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.390+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e040c5a70 con 0x7f2e04158a40 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.390+0000 7f2e01ffb640 1 -- 172.21.15.6:0/464114323 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc004bd0 con 0x7f2e04158a40 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.390+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e040c6310 con 0x7f2e04158a40 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.391+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e040c7f60 con 0x7f2e04158a40 2024-03-18T14:43:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.391+0000 7f2e01ffb640 1 -- 172.21.15.6:0/464114323 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2dfc011070 con 0x7f2e04158a40 2024-03-18T14:43:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.391+0000 7f2e01ffb640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2df006bbd0 0x7f2df006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.391+0000 7f2e01ffb640 1 -- 172.21.15.6:0/464114323 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7f2dfc08a420 con 0x7f2e04158a40 2024-03-18T14:43:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.391+0000 7f2e094d7640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2df006bbd0 0x7f2df006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.392+0000 7f2e094d7640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2df006bbd0 0x7f2df006e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009b20 tx=0x7f2df40039d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.396+0000 7f2e01ffb640 1 -- 172.21.15.6:0/464114323 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2dfc0590d0 con 0x7f2e04158a40 2024-03-18T14:43:39.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.577+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2e0409e970 con 0x7f2e04158a40 2024-03-18T14:43:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.955+0000 7f2e01ffb640 1 -- 172.21.15.6:0/464114323 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) ==== 138+0+0 (secure 0 0 0) 0x7f2dfc0589d0 con 0x7f2e04158a40 2024-03-18T14:43:39.958 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2df006bbd0 msgr2=0x7f2df006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2df006bbd0 0x7f2df006e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009b20 tx=0x7f2df40039d0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e04158a40 msgr2=0x7f2e0414aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e04158a40 0x7f2e0414aa00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c3e0 tx=0x7f2dfc00c8a0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 shutdown_connections 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f2df006bbd0 0x7f2df006e090 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e04158a40 0x7f2e0414aa00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e0415cc50 0x7f2e040af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 --2- 172.21.15.6:0/464114323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e040c4080 0x7f2e040ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:39.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 >> 172.21.15.6:0/464114323 conn(0x7f2e040d0980 msgr2=0x7f2e0415be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:39.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.957+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 shutdown_connections 2024-03-18T14:43:39.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:39.958+0000 7f2e0b73b640 1 -- 172.21.15.6:0/464114323 wait complete. 2024-03-18T14:43:39.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T14:43:40.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.081+0000 7f6cfffc4640 1 Processor -- start 2024-03-18T14:43:40.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.081+0000 7f6cfffc4640 1 -- start start 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.082+0000 7f6cfffc4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6cf80d0b60 0x7f6cf80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.082+0000 7f6cfffc4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf80d39d0 con 0x7f6cf80d3570 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.082+0000 7f6cfffc4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf80d2fd0 con 0x7f6cf815e030 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.082+0000 7f6cfffc4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf815e410 con 0x7f6cf80d0b60 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.082+0000 7f6cfd55f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6cf80d3570 0x7f6cf80d0450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57020/0 (socket says 172.21.15.6:57020) 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.082+0000 7f6cfd55f640 1 -- 172.21.15.6:0/1817384110 learned_addr learned my addr 172.21.15.6:0/1817384110 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.082+0000 7f6cfe561640 1 --2- 172.21.15.6:0/1817384110 >> v2:172.21.15.92:3300/0 conn(0x7f6cf80d0b60 0x7f6cf80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1848815733 0 0) 0x7f6cf80d39d0 con 0x7f6cf80d3570 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ce4003520 con 0x7f6cf80d3570 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2860714814 0 0) 0x7f6ce4003520 con 0x7f6cf80d3570 2024-03-18T14:43:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 >> v2:172.21.15.92:3300/0 conn(0x7f6cf80d0b60 msgr2=0x7f6cf80b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 --2- 172.21.15.6:0/1817384110 >> v2:172.21.15.92:3300/0 conn(0x7f6cf80d0b60 0x7f6cf80b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 >> v1:172.21.15.92:6789/0 conn(0x7f6cf815e030 legacy=0x7f6cf80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf80d3290 con 0x7f6cf80d3570 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f6ce8001670 con 0x7f6cf80d3570 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 >> v1:172.21.15.6:6789/0 conn(0x7f6cf80d3570 legacy=0x7f6cf80d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce40039b0 0x7f6ce4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce4004800 0x7f6ce4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6ce400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf815e410 con 0x7f6ce40051c0 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce400c060 con 0x7f6ce4004800 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce400c260 con 0x7f6ce40039b0 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfe561640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6ce400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfdd60640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce40039b0 0x7f6ce4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfd55f640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce4004800 0x7f6ce4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfe561640 1 -- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce40039b0 msgr2=0x7f6ce4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:40.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfe561640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce40039b0 0x7f6ce4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfe561640 1 -- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce4004800 msgr2=0x7f6ce4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:40.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfe561640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce4004800 0x7f6ce4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.083+0000 7f6cfe561640 1 -- 172.21.15.6:0/1817384110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce4003520 con 0x7f6ce40051c0 2024-03-18T14:43:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.084+0000 7f6cfe561640 1 -- 172.21.15.6:0/1817384110 reap_dead start 2024-03-18T14:43:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.084+0000 7f6cfe561640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6ce400b9c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ac0 tx=0x7f6ce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458dfaeac271d53a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.084+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4009690 con 0x7f6ce40051c0 2024-03-18T14:43:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.084+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4004020 con 0x7f6ce40051c0 2024-03-18T14:43:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.084+0000 7f6cfcd5e640 1 -- 172.21.15.6:0/1817384110 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4004410 con 0x7f6ce40051c0 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.085+0000 7f6cfffc4640 1 -- 172.21.15.6:0/1817384110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 msgr2=0x7f6ce400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.085+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6ce400b9c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ac0 tx=0x7f6ce4005830 comp rx=0 tx=0).stop 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.085+0000 7f6cfffc4640 1 -- 172.21.15.6:0/1817384110 shutdown_connections 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.085+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/1817384110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6ce400b9c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.085+0000 7f6cfffc4640 1 -- 172.21.15.6:0/1817384110 >> 172.21.15.6:0/1817384110 conn(0x7f6cf806c8c0 msgr2=0x7f6cf806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.085+0000 7f6cfffc4640 1 -- 172.21.15.6:0/1817384110 shutdown_connections 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.085+0000 7f6cfffc4640 1 -- 172.21.15.6:0/1817384110 wait complete. 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.086+0000 7f6cfffc4640 1 Processor -- start 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.086+0000 7f6cfffc4640 1 -- start start 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfffc4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6cf815f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfffc4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cf815fb20 0x7f6cf8149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfffc4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cf8149750 0x7f6cf8149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfffc4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf80ce830 con 0x7f6ce40051c0 2024-03-18T14:43:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfffc4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf80ce300 con 0x7f6cf815fb20 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfffc4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf80b6140 con 0x7f6cf8149750 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6cf815f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6cf815f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57498/0 (socket says 172.21.15.6:57498) 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 -- 172.21.15.6:0/977216259 learned_addr learned my addr 172.21.15.6:0/977216259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfd55f640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cf815fb20 0x7f6cf8149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfe561640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cf8149750 0x7f6cf8149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 -- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cf8149750 msgr2=0x7f6cf8149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cf8149750 0x7f6cf8149c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 -- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cf815fb20 msgr2=0x7f6cf8149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cf815fb20 0x7f6cf8149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 -- 172.21.15.6:0/977216259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf815c3f0 con 0x7f6ce40051c0 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfd55f640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cf815fb20 0x7f6cf8149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfe561640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cf8149750 0x7f6cf8149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.087+0000 7f6cfdd60640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6cf815f5e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000f960 tx=0x7f6cf000fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.088+0000 7f6cee7fc640 1 -- 172.21.15.6:0/977216259 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0003b80 con 0x7f6ce40051c0 2024-03-18T14:43:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.088+0000 7f6cee7fc640 1 -- 172.21.15.6:0/977216259 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf0018070 con 0x7f6ce40051c0 2024-03-18T14:43:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.088+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf81601c0 con 0x7f6ce40051c0 2024-03-18T14:43:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.088+0000 7f6cee7fc640 1 -- 172.21.15.6:0/977216259 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000d7f0 con 0x7f6ce40051c0 2024-03-18T14:43:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.088+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf8160780 con 0x7f6ce40051c0 2024-03-18T14:43:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.089+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf80751b0 con 0x7f6ce40051c0 2024-03-18T14:43:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.089+0000 7f6cee7fc640 1 -- 172.21.15.6:0/977216259 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6cf000d990 con 0x7f6ce40051c0 2024-03-18T14:43:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.090+0000 7f6cee7fc640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6cdc06bbd0 0x7f6cdc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.090+0000 7f6cee7fc640 1 -- 172.21.15.6:0/977216259 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5619+0+0 (secure 0 0 0) 0x7f6cf008ab50 con 0x7f6ce40051c0 2024-03-18T14:43:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.090+0000 7f6cfd55f640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6cdc06bbd0 0x7f6cdc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.090+0000 7f6cfd55f640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6cdc06bbd0 0x7f6cdc06e090 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8000fb0 tx=0x7f6ce8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.094+0000 7f6cee7fc640 1 -- 172.21.15.6:0/977216259 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6cf0059720 con 0x7f6ce40051c0 2024-03-18T14:43:40.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.279+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6cf80d19f0 con 0x7f6ce40051c0 2024-03-18T14:43:40.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.964+0000 7f6cee7fc640 1 -- 172.21.15.6:0/977216259 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) ==== 170+0+0 (secure 0 0 0) 0x7f6cf0059020 con 0x7f6ce40051c0 2024-03-18T14:43:40.966 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-18T14:43:40.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.968+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6cdc06bbd0 msgr2=0x7f6cdc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:40.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.968+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6cdc06bbd0 0x7f6cdc06e090 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8000fb0 tx=0x7f6ce8009040 comp rx=0 tx=0).stop 2024-03-18T14:43:40.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.968+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 msgr2=0x7f6cf815f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:40.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.968+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6cf815f5e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000f960 tx=0x7f6cf000fe20 comp rx=0 tx=0).stop 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.970+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 shutdown_connections 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.970+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6cdc06bbd0 0x7f6cdc06e090 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.970+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cf8149750 0x7f6cf8149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.970+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cf815fb20 0x7f6cf8149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.970+0000 7f6cfffc4640 1 --2- 172.21.15.6:0/977216259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce40051c0 0x7f6cf815f5e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.970+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 >> 172.21.15.6:0/977216259 conn(0x7f6cf806c8c0 msgr2=0x7f6cf80b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.971+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 shutdown_connections 2024-03-18T14:43:40.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:40.971+0000 7f6cfffc4640 1 -- 172.21.15.6:0/977216259 wait complete. 2024-03-18T14:43:40.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T14:43:41.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.104+0000 7fbfe5419640 1 Processor -- start 2024-03-18T14:43:41.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.104+0000 7fbfe5419640 1 -- start start 2024-03-18T14:43:41.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.104+0000 7fbfe5419640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbfe00b0ea0 0x7fbfe0165af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.104+0000 7fbfe5419640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe015f6c0 con 0x7fbfe0143ed0 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.104+0000 7fbfe5419640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe015e900 con 0x7fbfe01443c0 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.104+0000 7fbfe5419640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe00cf7d0 con 0x7fbfe00b0ea0 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfdeffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbfe0143ed0 0x7fbfe00cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57032/0 (socket says 172.21.15.6:57032) 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfdeffd640 1 -- 172.21.15.6:0/1511226709 learned_addr learned my addr 172.21.15.6:0/1511226709 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142749616 0 0) 0x7fbfe015f6c0 con 0x7fbfe0143ed0 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfc8003520 con 0x7fbfe0143ed0 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1511226709 >> v2:172.21.15.92:3300/0 conn(0x7fbfe00b0ea0 0x7fbfe0165af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 414722961 0 0) 0x7fbfc8003520 con 0x7fbfe0143ed0 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 >> v2:172.21.15.92:3300/0 conn(0x7fbfe00b0ea0 msgr2=0x7fbfe0165af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 --2- 172.21.15.6:0/1511226709 >> v2:172.21.15.92:3300/0 conn(0x7fbfe00b0ea0 0x7fbfe0165af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 >> v1:172.21.15.92:6789/0 conn(0x7fbfe01443c0 legacy=0x7fbfe00b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe015ebe0 con 0x7fbfe0143ed0 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fbfd0001670 con 0x7fbfe0143ed0 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 >> v1:172.21.15.6:6789/0 conn(0x7fbfe0143ed0 legacy=0x7fbfe00cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc80039b0 0x7fbfc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc8004800 0x7fbfc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc80051c0 0x7fbfc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc800c030 con 0x7fbfc80051c0 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc800c1e0 con 0x7fbfc80039b0 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc800c3e0 con 0x7fbfc8004800 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc80051c0 0x7fbfc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfdeffd640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc80039b0 0x7fbfc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.105+0000 7fbfde7fc640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc8004800 0x7fbfc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.106+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc8004800 msgr2=0x7fbfc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.106+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc8004800 0x7fbfc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.106+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc80039b0 msgr2=0x7fbfc8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.106+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc80039b0 0x7fbfc8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.106+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1511226709 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc8003520 con 0x7fbfc80051c0 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.106+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1511226709 reap_dead start 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.106+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc80051c0 0x7fbfc800b9c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00718b0 tx=0x7fbfc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e97edee56e376c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8009c10 con 0x7fbfc80051c0 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd802cbd0 con 0x7fbfc80051c0 2024-03-18T14:43:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfddffb640 1 -- 172.21.15.6:0/1511226709 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8003990 con 0x7fbfc80051c0 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfe5419640 1 -- 172.21.15.6:0/1511226709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc80051c0 msgr2=0x7fbfc800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc80051c0 0x7fbfc800b9c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00718b0 tx=0x7fbfc8005830 comp rx=0 tx=0).stop 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfe5419640 1 -- 172.21.15.6:0/1511226709 shutdown_connections 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1511226709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc80051c0 0x7fbfc800b9c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfe5419640 1 -- 172.21.15.6:0/1511226709 >> 172.21.15.6:0/1511226709 conn(0x7fbfe006b1c0 msgr2=0x7fbfe006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfe5419640 1 -- 172.21.15.6:0/1511226709 shutdown_connections 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.107+0000 7fbfe5419640 1 -- 172.21.15.6:0/1511226709 wait complete. 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 Processor -- start 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 -- start start 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc80051c0 0x7fbfe015a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe015d660 0x7fbfe015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfe00c5930 0x7fbfe00c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe00b0790 con 0x7fbfe00c5930 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe015f230 con 0x7fbfe015d660 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfe5419640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe00b0910 con 0x7fbfc80051c0 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfdf7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfe00c5930 0x7fbfe00c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfdf7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfe00c5930 0x7fbfe00c5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57514/0 (socket says 172.21.15.6:57514) 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.109+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1782524797 learned_addr learned my addr 172.21.15.6:0/1782524797 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdeffd640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc80051c0 0x7fbfe015a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfde7fc640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe015d660 0x7fbfe015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc80051c0 msgr2=0x7fbfe015a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc80051c0 0x7fbfe015a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe015d660 msgr2=0x7fbfe015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe015d660 0x7fbfe015dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdf7fe640 1 -- 172.21.15.6:0/1782524797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe015c5c0 con 0x7fbfe00c5930 2024-03-18T14:43:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdeffd640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc80051c0 0x7fbfe015a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfdf7fe640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfe00c5930 0x7fbfe00c5db0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80044a0 tx=0x7fbfd8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfc77fe640 1 -- 172.21.15.6:0/1782524797 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8003e10 con 0x7fbfe00c5930 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfc77fe640 1 -- 172.21.15.6:0/1782524797 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd8044070 con 0x7fbfe00c5930 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfc77fe640 1 -- 172.21.15.6:0/1782524797 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd803f470 con 0x7fbfe00c5930 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe015c8a0 con 0x7fbfe00c5930 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.110+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe015ce60 con 0x7fbfe00c5930 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.111+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe00751b0 con 0x7fbfe00c5930 2024-03-18T14:43:41.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.112+0000 7fbfc77fe640 1 -- 172.21.15.6:0/1782524797 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbfd8005d10 con 0x7fbfe00c5930 2024-03-18T14:43:41.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.112+0000 7fbfc77fe640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbfcc06bbd0 0x7fbfcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:41.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.112+0000 7fbfc77fe640 1 -- 172.21.15.6:0/1782524797 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6001+0+0 (secure 0 0 0) 0x7fbfd80ad1f0 con 0x7fbfe00c5930 2024-03-18T14:43:41.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.112+0000 7fbfdeffd640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbfcc06bbd0 0x7fbfcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:41.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.113+0000 7fbfdeffd640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbfcc06bbd0 0x7fbfcc06e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00014f0 tx=0x7fbfd0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:41.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.117+0000 7fbfc77fe640 1 -- 172.21.15.6:0/1782524797 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbfd807bc40 con 0x7fbfe00c5930 2024-03-18T14:43:41.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.336+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbfe00b3260 con 0x7fbfe00c5930 2024-03-18T14:43:41.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.979+0000 7fbfc77fe640 1 -- 172.21.15.6:0/1782524797 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v21) ==== 172+0+0 (secure 0 0 0) 0x7fbfd807b540 con 0x7fbfe00c5930 2024-03-18T14:43:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 8 and data pool 9 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbfcc06bbd0 msgr2=0x7fbfcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbfcc06bbd0 0x7fbfcc06e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00014f0 tx=0x7fbfd0009040 comp rx=0 tx=0).stop 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfe00c5930 msgr2=0x7fbfe00c5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfe00c5930 0x7fbfe00c5db0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80044a0 tx=0x7fbfd8009750 comp rx=0 tx=0).stop 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 shutdown_connections 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbfcc06bbd0 0x7fbfcc06e090 secure :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00014f0 tx=0x7fbfd0009040 comp rx=0 tx=0).stop 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfe00c5930 0x7fbfe00c5db0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe015d660 0x7fbfe015dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 --2- 172.21.15.6:0/1782524797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfc80051c0 0x7fbfe015a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.989+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 >> 172.21.15.6:0/1782524797 conn(0x7fbfe006b1c0 msgr2=0x7fbfe00ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.990+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 shutdown_connections 2024-03-18T14:43:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:41.990+0000 7fbfe5419640 1 -- 172.21.15.6:0/1782524797 wait complete. 2024-03-18T14:43:42.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:42.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.118+0000 7f09f4bb2640 1 Processor -- start 2024-03-18T14:43:42.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.118+0000 7f09f4bb2640 1 -- start start 2024-03-18T14:43:42.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.118+0000 7f09f4bb2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09f006cce0 0x7f09f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.118+0000 7f09f4bb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00b17b0 0x7f09f00b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.118+0000 7f09f4bb2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f00cfd80 con 0x7f09f00ce3c0 2024-03-18T14:43:42.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.118+0000 7f09f4bb2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f09f00b2df0 con 0x7f09f006cce0 2024-03-18T14:43:42.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.118+0000 7f09f4bb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f09f006d600 con 0x7f09f00b17b0 2024-03-18T14:43:42.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09ee59c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09f006cce0 0x7f09f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09edd9b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00b17b0 0x7f09f00b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09ee59c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09f006cce0 0x7f09f006d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40944/0 (socket says 172.21.15.6:40944) 2024-03-18T14:43:42.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09edd9b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00b17b0 0x7f09f00b1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51538/0 (socket says 172.21.15.6:51538) 2024-03-18T14:43:42.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09ee59c640 1 -- 172.21.15.6:0/3063935812 learned_addr learned my addr 172.21.15.6:0/3063935812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:42.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09edd9b640 1 -- 172.21.15.6:0/3063935812 >> v2:172.21.15.92:3300/0 conn(0x7f09f006cce0 msgr2=0x7f09f006d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:42.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09edd9b640 1 --2- 172.21.15.6:0/3063935812 >> v2:172.21.15.92:3300/0 conn(0x7f09f006cce0 0x7f09f006d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09edd9b640 1 -- 172.21.15.6:0/3063935812 >> v1:172.21.15.92:6789/0 conn(0x7f09f00ce3c0 legacy=0x7f09f00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09edd9b640 1 -- 172.21.15.6:0/3063935812 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00b2270 con 0x7f09f00b17b0 2024-03-18T14:43:42.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.119+0000 7f09ee59c640 1 --2- 172.21.15.6:0/3063935812 >> v2:172.21.15.92:3300/0 conn(0x7f09f006cce0 0x7f09f006d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:42.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.120+0000 7f09edd9b640 1 --2- 172.21.15.6:0/3063935812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00b17b0 0x7f09f00b1c10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09d80097a0 tx=0x7f09d802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17f996e3265c6282 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.120+0000 7f09ed59a640 1 -- 172.21.15.6:0/3063935812 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d80315a0 con 0x7f09f00b17b0 2024-03-18T14:43:42.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.120+0000 7f09ed59a640 1 -- 172.21.15.6:0/3063935812 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d8037070 con 0x7f09f00b17b0 2024-03-18T14:43:42.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.120+0000 7f09ed59a640 1 -- 172.21.15.6:0/3063935812 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8031960 con 0x7f09f00b17b0 2024-03-18T14:43:42.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 -- 172.21.15.6:0/3063935812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00b17b0 msgr2=0x7f09f00b1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/3063935812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00b17b0 0x7f09f00b1c10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09d80097a0 tx=0x7f09d802b840 comp rx=0 tx=0).stop 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 -- 172.21.15.6:0/3063935812 shutdown_connections 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/3063935812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00b17b0 0x7f09f00b1c10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/3063935812 >> v2:172.21.15.92:3300/0 conn(0x7f09f006cce0 0x7f09f006d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 -- 172.21.15.6:0/3063935812 >> 172.21.15.6:0/3063935812 conn(0x7f09f008b130 msgr2=0x7f09f008b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 -- 172.21.15.6:0/3063935812 shutdown_connections 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.121+0000 7f09f4bb2640 1 -- 172.21.15.6:0/3063935812 wait complete. 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.122+0000 7f09f4bb2640 1 Processor -- start 2024-03-18T14:43:42.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.122+0000 7f09f4bb2640 1 -- start start 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09f4bb2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09f006cce0 0x7f09f00b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09f4bb2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b17b0 0x7f09f00b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09f4bb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00ce3c0 0x7f09f00b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09f4bb2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00b2b10 con 0x7f09f006cce0 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09f4bb2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00b2830 con 0x7f09f00b17b0 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09f4bb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f09f015e330 con 0x7f09f00ce3c0 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09f006cce0 0x7f09f00b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09f006cce0 0x7f09f00b6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57520/0 (socket says 172.21.15.6:57520) 2024-03-18T14:43:42.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 -- 172.21.15.6:0/309978078 learned_addr learned my addr 172.21.15.6:0/309978078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09edd9b640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b17b0 0x7f09f00b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09eed9d640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00ce3c0 0x7f09f00b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 -- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00ce3c0 msgr2=0x7f09f00b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00ce3c0 0x7f09f00b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 -- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b17b0 msgr2=0x7f09f00b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b17b0 0x7f09f00b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.123+0000 7f09ee59c640 1 -- 172.21.15.6:0/309978078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00b7610 con 0x7f09f006cce0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.124+0000 7f09ee59c640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09f006cce0 0x7f09f00b6fc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f09e800e9e0 tx=0x7f09e800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.124+0000 7f09e6ffd640 1 -- 172.21.15.6:0/309978078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8017070 con 0x7f09f006cce0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.124+0000 7f09e6ffd640 1 -- 172.21.15.6:0/309978078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e800c9d0 con 0x7f09f006cce0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.124+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f00b6150 con 0x7f09f006cce0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.124+0000 7f09e6ffd640 1 -- 172.21.15.6:0/309978078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e800cd80 con 0x7f09f006cce0 2024-03-18T14:43:42.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.124+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f00b66d0 con 0x7f09f006cce0 2024-03-18T14:43:42.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.125+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f00b0d20 con 0x7f09f006cce0 2024-03-18T14:43:42.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.125+0000 7f09e6ffd640 1 -- 172.21.15.6:0/309978078 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f09e801a070 con 0x7f09f006cce0 2024-03-18T14:43:42.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.126+0000 7f09e6ffd640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f09d006bbd0 0x7f09d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.126+0000 7f09e6ffd640 1 -- 172.21.15.6:0/309978078 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f09e808aa40 con 0x7f09f006cce0 2024-03-18T14:43:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.126+0000 7f09edd9b640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f09d006bbd0 0x7f09d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.127+0000 7f09edd9b640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f09d006bbd0 0x7f09d006e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f09d8036000 tx=0x7f09d803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.130+0000 7f09e6ffd640 1 -- 172.21.15.6:0/309978078 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f09e8058d60 con 0x7f09f006cce0 2024-03-18T14:43:42.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.315+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09f009fa80 con 0x7f09f006cce0 2024-03-18T14:43:42.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.316+0000 7f09e6ffd640 1 -- 172.21.15.6:0/309978078 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16855 (secure 0 0 0) 0x7f09e805ce00 con 0x7f09f006cce0 2024-03-18T14:43:42.318 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:42.319 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":55,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:41.338428+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f09d006bbd0 msgr2=0x7f09d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f09d006bbd0 0x7f09d006e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f09d8036000 tx=0x7f09d803d040 comp rx=0 tx=0).stop 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09f006cce0 msgr2=0x7f09f00b6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09f006cce0 0x7f09f00b6fc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f09e800e9e0 tx=0x7f09e800eea0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 shutdown_connections 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f09d006bbd0 0x7f09d006e090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09f00ce3c0 0x7f09f00b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b17b0 0x7f09f00b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 --2- 172.21.15.6:0/309978078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09f006cce0 0x7f09f00b6fc0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 >> 172.21.15.6:0/309978078 conn(0x7f09f008b130 msgr2=0x7f09f00d1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 shutdown_connections 2024-03-18T14:43:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.318+0000 7f09f4bb2640 1 -- 172.21.15.6:0/309978078 wait complete. 2024-03-18T14:43:42.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:42.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.430+0000 7f5e2af59640 1 Processor -- start 2024-03-18T14:43:42.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.430+0000 7f5e2af59640 1 -- start start 2024-03-18T14:43:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e2af59640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e2415e020 0x7f5e24162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e2af59640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240d3210 0x7f5e240d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e2af59640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e240b9660 con 0x7f5e240ced00 2024-03-18T14:43:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e2af59640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5e240d2270 con 0x7f5e2415e020 2024-03-18T14:43:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e2af59640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5e240ce390 con 0x7f5e240d3210 2024-03-18T14:43:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e28cf5640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e2415e020 0x7f5e24162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e28cf5640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e2415e020 0x7f5e24162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57522/0 (socket says 172.21.15.6:57522) 2024-03-18T14:43:42.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e28cf5640 1 -- 172.21.15.6:0/3375091783 learned_addr learned my addr 172.21.15.6:0/3375091783 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:42.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.431+0000 7f5e294f6640 1 --2- 172.21.15.6:0/3375091783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240d3210 0x7f5e240d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.432+0000 7f5e28cf5640 1 -- 172.21.15.6:0/3375091783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240d3210 msgr2=0x7f5e240d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:42.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.432+0000 7f5e28cf5640 1 --2- 172.21.15.6:0/3375091783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240d3210 0x7f5e240d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.432+0000 7f5e28cf5640 1 -- 172.21.15.6:0/3375091783 >> v1:172.21.15.92:6789/0 conn(0x7f5e240ced00 legacy=0x7f5e240d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.432+0000 7f5e28cf5640 1 -- 172.21.15.6:0/3375091783 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e240b9900 con 0x7f5e2415e020 2024-03-18T14:43:42.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.432+0000 7f5e294f6640 1 --2- 172.21.15.6:0/3375091783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240d3210 0x7f5e240d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.432+0000 7f5e28cf5640 1 --2- 172.21.15.6:0/3375091783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24162380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5e240b9830 tx=0x7f5e1c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de51db1626cca20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e237fe640 1 -- 172.21.15.6:0/3375091783 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c02ca50 con 0x7f5e2415e020 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e237fe640 1 -- 172.21.15.6:0/3375091783 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1c02cbf0 con 0x7f5e2415e020 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e237fe640 1 -- 172.21.15.6:0/3375091783 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0343f0 con 0x7f5e2415e020 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 -- 172.21.15.6:0/3375091783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 msgr2=0x7f5e24162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 --2- 172.21.15.6:0/3375091783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24162380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5e240b9830 tx=0x7f5e1c02beb0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 -- 172.21.15.6:0/3375091783 shutdown_connections 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 --2- 172.21.15.6:0/3375091783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240d3210 0x7f5e240d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 --2- 172.21.15.6:0/3375091783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24162380 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 -- 172.21.15.6:0/3375091783 >> 172.21.15.6:0/3375091783 conn(0x7f5e24072ce0 msgr2=0x7f5e240730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:42.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 -- 172.21.15.6:0/3375091783 shutdown_connections 2024-03-18T14:43:42.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.433+0000 7f5e2af59640 1 -- 172.21.15.6:0/3375091783 wait complete. 2024-03-18T14:43:42.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.434+0000 7f5e2af59640 1 Processor -- start 2024-03-18T14:43:42.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.434+0000 7f5e2af59640 1 -- start start 2024-03-18T14:43:42.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e2af59640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240ced00 0x7f5e24154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e2af59640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e240d3210 0x7f5e24151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e2af59640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e2af59640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240cf2c0 con 0x7f5e2415e020 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e2af59640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240ce890 con 0x7f5e240d3210 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e2af59640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5e240cf440 con 0x7f5e240ced00 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e294f6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e294f6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57532/0 (socket says 172.21.15.6:57532) 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e294f6640 1 -- 172.21.15.6:0/1381091428 learned_addr learned my addr 172.21.15.6:0/1381091428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e23fff640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e240d3210 0x7f5e24151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.435+0000 7f5e28cf5640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240ced00 0x7f5e24154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e294f6640 1 -- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240ced00 msgr2=0x7f5e24154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e294f6640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240ced00 0x7f5e24154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e294f6640 1 -- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e240d3210 msgr2=0x7f5e24151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e294f6640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e240d3210 0x7f5e24151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e294f6640 1 -- 172.21.15.6:0/1381091428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e24153de0 con 0x7f5e2415e020 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e23fff640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e240d3210 0x7f5e24151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e28cf5640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240ced00 0x7f5e24154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e294f6640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24153400 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400e7e0 tx=0x7f5e1400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e217fa640 1 -- 172.21.15.6:0/1381091428 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1400f040 con 0x7f5e2415e020 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e217fa640 1 -- 172.21.15.6:0/1381091428 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e14004030 con 0x7f5e2415e020 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e241540c0 con 0x7f5e2415e020 2024-03-18T14:43:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e217fa640 1 -- 172.21.15.6:0/1381091428 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e140043e0 con 0x7f5e2415e020 2024-03-18T14:43:42.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.436+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e24154680 con 0x7f5e2415e020 2024-03-18T14:43:42.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.437+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2408afe0 con 0x7f5e2415e020 2024-03-18T14:43:42.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.437+0000 7f5e217fa640 1 -- 172.21.15.6:0/1381091428 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5e1400c6d0 con 0x7f5e2415e020 2024-03-18T14:43:42.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.438+0000 7f5e217fa640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e1006bc00 0x7f5e1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.438+0000 7f5e217fa640 1 -- 172.21.15.6:0/1381091428 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f5e1408ac20 con 0x7f5e2415e020 2024-03-18T14:43:42.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.438+0000 7f5e28cf5640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e1006bc00 0x7f5e1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.438+0000 7f5e28cf5640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e1006bc00 0x7f5e1006e0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c035000 tx=0x7f5e1c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.442+0000 7f5e217fa640 1 -- 172.21.15.6:0/1381091428 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5e14017020 con 0x7f5e2415e020 2024-03-18T14:43:42.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.626+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e2404f620 con 0x7f5e2415e020 2024-03-18T14:43:42.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.627+0000 7f5e217fa640 1 -- 172.21.15.6:0/1381091428 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16855 (secure 0 0 0) 0x7f5e14059580 con 0x7f5e2415e020 2024-03-18T14:43:42.629 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:42.629 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":55,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:41.338428+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:42.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e1006bc00 msgr2=0x7f5e1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e1006bc00 0x7f5e1006e0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c035000 tx=0x7f5e1c030000 comp rx=0 tx=0).stop 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 msgr2=0x7f5e24153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24153400 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400e7e0 tx=0x7f5e1400eca0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 shutdown_connections 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e1006bc00 0x7f5e1006e0c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2415e020 0x7f5e24153400 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e240d3210 0x7f5e24151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 --2- 172.21.15.6:0/1381091428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e240ced00 0x7f5e24154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 >> 172.21.15.6:0/1381091428 conn(0x7f5e24072ce0 msgr2=0x7f5e240d2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 shutdown_connections 2024-03-18T14:43:42.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.629+0000 7f5e2af59640 1 -- 172.21.15.6:0/1381091428 wait complete. 2024-03-18T14:43:42.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T14:43:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.745+0000 7f23c13c6640 1 Processor -- start 2024-03-18T14:43:42.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.745+0000 7f23c13c6640 1 -- start start 2024-03-18T14:43:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.746+0000 7f23c13c6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc0cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.746+0000 7f23c13c6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f23bc0d2510 0x7f23bc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.746+0000 7f23c13c6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23bc0d2960 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.746+0000 7f23c13c6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f23bc0cd8d0 con 0x7f23bc0d2510 2024-03-18T14:43:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.746+0000 7f23c13c6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f23bc0cfac0 con 0x7f23bc0d0730 2024-03-18T14:43:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.746+0000 7f23baffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f23bc0ce9d0 0x7f23bc0d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57034/0 (socket says 172.21.15.6:57034) 2024-03-18T14:43:42.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.746+0000 7f23baffd640 1 -- 172.21.15.6:0/304696339 learned_addr learned my addr 172.21.15.6:0/304696339 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23b27fc640 1 --2- 172.21.15.6:0/304696339 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc0cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23bb7fe640 1 --2- 172.21.15.6:0/304696339 >> v2:172.21.15.92:3300/0 conn(0x7f23bc0d2510 0x7f23bc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23ba7fc640 1 -- 172.21.15.6:0/304696339 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1335940467 0 0) 0x7f23bc0d2960 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23ba7fc640 1 -- 172.21.15.6:0/304696339 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23a8003520 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23b27fc640 1 -- 172.21.15.6:0/304696339 >> v2:172.21.15.92:3300/0 conn(0x7f23bc0d2510 msgr2=0x7f23bc15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23b27fc640 1 --2- 172.21.15.6:0/304696339 >> v2:172.21.15.92:3300/0 conn(0x7f23bc0d2510 0x7f23bc15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23b27fc640 1 -- 172.21.15.6:0/304696339 >> v1:172.21.15.6:6789/0 conn(0x7f23bc0ce9d0 legacy=0x7f23bc0d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23ba7fc640 1 -- 172.21.15.6:0/304696339 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2962755993 0 0) 0x7f23a8003520 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.747+0000 7f23b27fc640 1 -- 172.21.15.6:0/304696339 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23bc0cccc0 con 0x7f23bc0d0730 2024-03-18T14:43:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23b27fc640 1 --2- 172.21.15.6:0/304696339 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc0cc650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f23a40097a0 tx=0x7f23a402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd2632a3b8f75e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23ba7fc640 1 -- 172.21.15.6:0/304696339 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4037070 con 0x7f23bc0d0730 2024-03-18T14:43:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23ba7fc640 1 -- 172.21.15.6:0/304696339 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23a4030510 con 0x7f23bc0d0730 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23ba7fc640 1 -- 172.21.15.6:0/304696339 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0017f0 con 0x7f23bc0d0730 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23c13c6640 1 -- 172.21.15.6:0/304696339 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 msgr2=0x7f23bc0cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23c13c6640 1 --2- 172.21.15.6:0/304696339 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc0cc650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f23a40097a0 tx=0x7f23a402b710 comp rx=0 tx=0).stop 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23c13c6640 1 -- 172.21.15.6:0/304696339 shutdown_connections 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23c13c6640 1 --2- 172.21.15.6:0/304696339 >> v2:172.21.15.92:3300/0 conn(0x7f23bc0d2510 0x7f23bc15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23c13c6640 1 --2- 172.21.15.6:0/304696339 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc0cc650 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23c13c6640 1 -- 172.21.15.6:0/304696339 >> 172.21.15.6:0/304696339 conn(0x7f23bc08b0c0 msgr2=0x7f23bc0d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.748+0000 7f23c13c6640 1 -- 172.21.15.6:0/304696339 shutdown_connections 2024-03-18T14:43:42.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.749+0000 7f23c13c6640 1 -- 172.21.15.6:0/304696339 wait complete. 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 Processor -- start 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 -- start start 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc0ce9d0 0x7f23bc0c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc0d2510 0x7f23bc14a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc0cd560 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc0cd280 con 0x7f23bc0d2510 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.750+0000 7f23c13c6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f23bc0cedb0 con 0x7f23bc0d0730 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc0ce9d0 0x7f23bc0c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc0ce9d0 0x7f23bc0c6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57542/0 (socket says 172.21.15.6:57542) 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 -- 172.21.15.6:0/3915437276 learned_addr learned my addr 172.21.15.6:0/3915437276 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23b27fc640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23bb7fe640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc0d2510 0x7f23bc14a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 -- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 msgr2=0x7f23bc149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 -- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc0d2510 msgr2=0x7f23bc14a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc0d2510 0x7f23bc14a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 -- 172.21.15.6:0/3915437276 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23bc0c5f80 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23bb7fe640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc0d2510 0x7f23bc14a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23baffd640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc0ce9d0 0x7f23bc0c6a20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c5a0 tx=0x7f23ac00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23b3fff640 1 -- 172.21.15.6:0/3915437276 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac017070 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23b3fff640 1 -- 172.21.15.6:0/3915437276 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23ac007480 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23b3fff640 1 -- 172.21.15.6:0/3915437276 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0059e0 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23bc0c8ed0 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.751+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23bc0c6220 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.752+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23bc0751b0 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.753+0000 7f23b3fff640 1 -- 172.21.15.6:0/3915437276 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f23ac01a070 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.753+0000 7f23b3fff640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f239c06bbd0 0x7f239c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.753+0000 7f23b3fff640 1 -- 172.21.15.6:0/3915437276 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f23ac08a090 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.753+0000 7f23b27fc640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f239c06bbd0 0x7f239c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.754+0000 7f23b27fc640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f239c06bbd0 0x7f239c06e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f23a4034000 tx=0x7f23a4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:42.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.757+0000 7f23b3fff640 1 -- 172.21.15.6:0/3915437276 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f23ac058a80 con 0x7f23bc0ce9d0 2024-03-18T14:43:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:42.986+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f23bc068be0 con 0x7f23bc0ce9d0 2024-03-18T14:43:43.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.986+0000 7f23b3fff640 1 -- 172.21.15.6:0/3915437276 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f23ac058380 con 0x7f23bc0ce9d0 2024-03-18T14:43:43.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f239c06bbd0 msgr2=0x7f239c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:43.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f239c06bbd0 0x7f239c06e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f23a4034000 tx=0x7f23a4008030 comp rx=0 tx=0).stop 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc0ce9d0 msgr2=0x7f23bc0c6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc0ce9d0 0x7f23bc0c6a20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c5a0 tx=0x7f23ac00ca60 comp rx=0 tx=0).stop 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 shutdown_connections 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f239c06bbd0 0x7f239c06e090 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc0d2510 0x7f23bc14a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc0d0730 0x7f23bc149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 --2- 172.21.15.6:0/3915437276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc0ce9d0 0x7f23bc0c6a20 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 >> 172.21.15.6:0/3915437276 conn(0x7f23bc08b0c0 msgr2=0x7f23bc0d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 shutdown_connections 2024-03-18T14:43:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:43.989+0000 7f23c13c6640 1 -- 172.21.15.6:0/3915437276 wait complete. 2024-03-18T14:43:44.002 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T14:43:44.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T14:43:44.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.108+0000 7f669a8f1640 1 Processor -- start 2024-03-18T14:43:44.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.108+0000 7f669a8f1640 1 -- start start 2024-03-18T14:43:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f669a8f1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66940b1ea0 0x7f66940ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:44.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f669a8f1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f66940d14d0 0x7f66940cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f669a8f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66940cc540 con 0x7f66940cf4d0 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f669a8f1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f66940cf920 con 0x7f66940b1ea0 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f669a8f1640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f66940b2280 con 0x7f66940d14d0 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f6693fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66940b1ea0 0x7f66940ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f6693fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66940b1ea0 0x7f66940ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35976/0 (socket says 172.21.15.6:35976) 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f6693fff640 1 -- 172.21.15.6:0/2696147501 learned_addr learned my addr 172.21.15.6:0/2696147501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f66937fe640 1 --2- 172.21.15.6:0/2696147501 >> v2:172.21.15.92:3300/0 conn(0x7f66940d14d0 0x7f66940cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f6693fff640 1 -- 172.21.15.6:0/2696147501 >> v2:172.21.15.92:3300/0 conn(0x7f66940d14d0 msgr2=0x7f66940cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f6693fff640 1 --2- 172.21.15.6:0/2696147501 >> v2:172.21.15.92:3300/0 conn(0x7f66940d14d0 0x7f66940cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f6693fff640 1 -- 172.21.15.6:0/2696147501 >> v1:172.21.15.6:6789/0 conn(0x7f66940cf4d0 legacy=0x7f669415e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:44.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.109+0000 7f6693fff640 1 -- 172.21.15.6:0/2696147501 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66940ceb60 con 0x7f66940b1ea0 2024-03-18T14:43:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.110+0000 7f6693fff640 1 --2- 172.21.15.6:0/2696147501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940b1ea0 0x7f66940ce580 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f66940d0630 tx=0x7f668402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644f881531a1c435 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:44.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.110+0000 7f6692ffd640 1 -- 172.21.15.6:0/2696147501 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684034070 con 0x7f66940b1ea0 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.110+0000 7f6692ffd640 1 -- 172.21.15.6:0/2696147501 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6684030860 con 0x7f66940b1ea0 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.110+0000 7f6692ffd640 1 -- 172.21.15.6:0/2696147501 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684030c30 con 0x7f66940b1ea0 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 -- 172.21.15.6:0/2696147501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940b1ea0 msgr2=0x7f66940ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 --2- 172.21.15.6:0/2696147501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940b1ea0 0x7f66940ce580 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f66940d0630 tx=0x7f668402b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 -- 172.21.15.6:0/2696147501 shutdown_connections 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 --2- 172.21.15.6:0/2696147501 >> v2:172.21.15.92:3300/0 conn(0x7f66940d14d0 0x7f66940cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 --2- 172.21.15.6:0/2696147501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940b1ea0 0x7f66940ce580 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 -- 172.21.15.6:0/2696147501 >> 172.21.15.6:0/2696147501 conn(0x7f66940d09b0 msgr2=0x7f66940d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 -- 172.21.15.6:0/2696147501 shutdown_connections 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.111+0000 7f669a8f1640 1 -- 172.21.15.6:0/2696147501 wait complete. 2024-03-18T14:43:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.113+0000 7f669a8f1640 1 Processor -- start 2024-03-18T14:43:44.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.113+0000 7f669a8f1640 1 -- start start 2024-03-18T14:43:44.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f669a8f1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66940b1ea0 0x7f6694153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:44.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f669a8f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f66940cf4d0 0x7f669406fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:44.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f669a8f1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940d14d0 0x7f6694070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f669a8f1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f669415b9a0 con 0x7f66940d14d0 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f669a8f1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f669415b6c0 con 0x7f66940b1ea0 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f669a8f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f66940cdcf0 con 0x7f66940cf4d0 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6698e8e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940d14d0 0x7f6694070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6698e8e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940d14d0 0x7f6694070280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35984/0 (socket says 172.21.15.6:35984) 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6698e8e640 1 -- 172.21.15.6:0/1909545500 learned_addr learned my addr 172.21.15.6:0/1909545500 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6693fff640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66940b1ea0 0x7f6694153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6698e8e640 1 -- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f66940cf4d0 msgr2=0x7f669406fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6698e8e640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f66940cf4d0 0x7f669406fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6698e8e640 1 -- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66940b1ea0 msgr2=0x7f6694153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.114+0000 7f6698e8e640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66940b1ea0 0x7f6694153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.115+0000 7f6698e8e640 1 -- 172.21.15.6:0/1909545500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669409e400 con 0x7f66940d14d0 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.115+0000 7f6698e8e640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940d14d0 0x7f6694070280 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f668c00c7d0 tx=0x7f668c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.115+0000 7f6690ff9640 1 -- 172.21.15.6:0/1909545500 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c003b70 con 0x7f66940d14d0 2024-03-18T14:43:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.115+0000 7f6690ff9640 1 -- 172.21.15.6:0/1909545500 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c003d30 con 0x7f66940d14d0 2024-03-18T14:43:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.115+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66940ad750 con 0x7f66940d14d0 2024-03-18T14:43:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.115+0000 7f6690ff9640 1 -- 172.21.15.6:0/1909545500 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c00aa90 con 0x7f66940d14d0 2024-03-18T14:43:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.115+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66940ad9f0 con 0x7f66940d14d0 2024-03-18T14:43:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.116+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66940751b0 con 0x7f66940d14d0 2024-03-18T14:43:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.116+0000 7f6690ff9640 1 -- 172.21.15.6:0/1909545500 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f668c00ac30 con 0x7f66940d14d0 2024-03-18T14:43:44.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.117+0000 7f6690ff9640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f668006bbd0 0x7f668006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:44.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.117+0000 7f6690ff9640 1 -- 172.21.15.6:0/1909545500 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f668c08aaa0 con 0x7f66940d14d0 2024-03-18T14:43:44.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.117+0000 7f6693fff640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f668006bbd0 0x7f668006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:44.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.118+0000 7f6693fff640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f668006bbd0 0x7f668006e090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6684036000 tx=0x7f668403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:44.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.121+0000 7f6690ff9640 1 -- 172.21.15.6:0/1909545500 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f668c059490 con 0x7f66940d14d0 2024-03-18T14:43:44.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.343+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f66940cdf60 con 0x7f66940d14d0 2024-03-18T14:43:44.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.990+0000 7f6690ff9640 1 -- 172.21.15.6:0/1909545500 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7f668c058d90 con 0x7f66940d14d0 2024-03-18T14:43:44.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f668006bbd0 msgr2=0x7f668006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:44.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f668006bbd0 0x7f668006e090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6684036000 tx=0x7f668403b040 comp rx=0 tx=0).stop 2024-03-18T14:43:44.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940d14d0 msgr2=0x7f6694070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940d14d0 0x7f6694070280 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f668c00c7d0 tx=0x7f668c00cc90 comp rx=0 tx=0).stop 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 shutdown_connections 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f668006bbd0 0x7f668006e090 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66940d14d0 0x7f6694070280 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f66940cf4d0 0x7f669406fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 --2- 172.21.15.6:0/1909545500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66940b1ea0 0x7f6694153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 >> 172.21.15.6:0/1909545500 conn(0x7f66940d09b0 msgr2=0x7f66940b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 shutdown_connections 2024-03-18T14:43:44.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:44.993+0000 7f669a8f1640 1 -- 172.21.15.6:0/1909545500 wait complete. 2024-03-18T14:43:45.006 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T14:43:45.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:45.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.109+0000 7f748c9c6640 1 Processor -- start 2024-03-18T14:43:45.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.110+0000 7f748c9c6640 1 -- start start 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.110+0000 7f748c9c6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f748815def0 0x7f74880cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.110+0000 7f748c9c6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 0x7f748806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.110+0000 7f748c9c6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74880b28b0 con 0x7f748806cce0 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.110+0000 7f748c9c6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f74880d0540 con 0x7f748815def0 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.110+0000 7f748c9c6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f74880b2b90 con 0x7f74880b2260 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f7485d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f748806cce0 0x7f74880b1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58076/0 (socket says 172.21.15.6:58076) 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f7485d9b640 1 -- 172.21.15.6:0/3857147585 learned_addr learned my addr 172.21.15.6:0/3857147585 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748659c640 1 --2- 172.21.15.6:0/3857147585 >> v2:172.21.15.92:3300/0 conn(0x7f748815def0 0x7f74880cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f7486d9d640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 0x7f748806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916194766 0 0) 0x7f74880b28b0 con 0x7f748806cce0 2024-03-18T14:43:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7470003520 con 0x7f748806cce0 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3294089982 0 0) 0x7f7470003520 con 0x7f748806cce0 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 msgr2=0x7f748806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 0x7f748806d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 >> v2:172.21.15.92:3300/0 conn(0x7f748815def0 msgr2=0x7f74880cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 --2- 172.21.15.6:0/3857147585 >> v2:172.21.15.92:3300/0 conn(0x7f748815def0 0x7f74880cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74880d01d0 con 0x7f748806cce0 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f747c001670 con 0x7f748806cce0 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 >> v1:172.21.15.6:6789/0 conn(0x7f748806cce0 legacy=0x7f74880b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74700039b0 0x7f7470003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f7470004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 0x7f74700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f74880d0540 con 0x7f7470004800 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74880b2b90 con 0x7f74700039b0 2024-03-18T14:43:45.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f747000b9c0 con 0x7f74880b2260 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f7470004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f748659c640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74700039b0 0x7f7470003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.111+0000 7f7486d9d640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 0x7f74700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.112+0000 7f7485d9b640 1 -- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 msgr2=0x7f74700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.112+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 0x7f74700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.112+0000 7f7485d9b640 1 -- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74700039b0 msgr2=0x7f7470003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.112+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74700039b0 0x7f7470003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.112+0000 7f7485d9b640 1 -- 172.21.15.6:0/3857147585 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7470003520 con 0x7f7470004800 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.112+0000 7f7486d9d640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74880b2260 0x7f74700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.112+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f7470004c80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f747c007fd0 tx=0x7f7470005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaed50d4d8fa1faf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c007d50 con 0x7f7470004800 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f747c030070 con 0x7f7470004800 2024-03-18T14:43:45.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748559a640 1 -- 172.21.15.6:0/3857147585 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c02cd40 con 0x7f7470004800 2024-03-18T14:43:45.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748c9c6640 1 -- 172.21.15.6:0/3857147585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 msgr2=0x7f7470004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748c9c6640 1 --2- 172.21.15.6:0/3857147585 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f7470004c80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f747c007fd0 tx=0x7f7470005830 comp rx=0 tx=0).stop 2024-03-18T14:43:45.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f7486d9d640 1 -- 172.21.15.6:0/3857147585 reap_dead start 2024-03-18T14:43:45.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748c9c6640 1 -- 172.21.15.6:0/3857147585 shutdown_connections 2024-03-18T14:43:45.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748c9c6640 1 -- 172.21.15.6:0/3857147585 >> 172.21.15.6:0/3857147585 conn(0x7f748809ae20 msgr2=0x7f748808b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748c9c6640 1 -- 172.21.15.6:0/3857147585 shutdown_connections 2024-03-18T14:43:45.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.113+0000 7f748c9c6640 1 -- 172.21.15.6:0/3857147585 wait complete. 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 Processor -- start 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 -- start start 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f748815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7488160b60 0x7f7488151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488151c40 0x7f74880afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f748806d160 con 0x7f7470004800 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f748815e2d0 con 0x7f7488151c40 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748c9c6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f748806d2e0 con 0x7f7488160b60 2024-03-18T14:43:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f748815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f748815e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36000/0 (socket says 172.21.15.6:36000) 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.115+0000 7f748659c640 1 -- 172.21.15.6:0/3019492515 learned_addr learned my addr 172.21.15.6:0/3019492515 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f7486d9d640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488151c40 0x7f74880afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7488160b60 0x7f7488151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748659c640 1 -- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7488160b60 msgr2=0x7f7488151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748659c640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7488160b60 0x7f7488151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748659c640 1 -- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488151c40 msgr2=0x7f74880afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748659c640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488151c40 0x7f74880afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748659c640 1 -- 172.21.15.6:0/3019492515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74880af590 con 0x7f7470004800 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f7486d9d640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488151c40 0x7f74880afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7488160b60 0x7f7488151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:45.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748659c640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f748815e6f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7480011990 tx=0x7f7480011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f7476ffd640 1 -- 172.21.15.6:0/3019492515 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748000fad0 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f7476ffd640 1 -- 172.21.15.6:0/3019492515 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f748000fc70 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f748815ec30 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f7476ffd640 1 -- 172.21.15.6:0/3019492515 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480014440 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.116+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74880c8dc0 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.117+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f748809fa90 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.118+0000 7f7476ffd640 1 -- 172.21.15.6:0/3019492515 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7480012040 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.118+0000 7f7476ffd640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f746806bbd0 0x7f746806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.118+0000 7f7476ffd640 1 -- 172.21.15.6:0/3019492515 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f748008b340 con 0x7f7470004800 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.118+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f746806bbd0 0x7f746806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.119+0000 7f7485d9b640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f746806bbd0 0x7f746806e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f747c034000 tx=0x7f747c007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.122+0000 7f7476ffd640 1 -- 172.21.15.6:0/3019492515 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f74880af590 con 0x7f7470004800 2024-03-18T14:43:45.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.341+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74880d2280 con 0x7f7470004800 2024-03-18T14:43:45.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.342+0000 7f7476ffd640 1 -- 172.21.15.6:0/3019492515 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f7480019030 con 0x7f7470004800 2024-03-18T14:43:45.344 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:45.344 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/1499402578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1499402578},{"type":"v1","addr":"172.21.15.92:6836","nonce":1499402578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6022,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/300544617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":300544617},{"type":"v1","addr":"172.21.15.6:6835","nonce":300544617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/175151750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":175151750},{"type":"v1","addr":"172.21.15.92:6837","nonce":175151750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:43:41.338763+0000","modified":"2024-03-18T14:43:44.344988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6225},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6225":{"gid":6225,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/3070832840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3070832840},{"type":"v1","addr":"172.21.15.6:6834","nonce":3070832840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6225,"qdb_cluster":[6225]},"id":3}]} 2024-03-18T14:43:45.345 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-18T14:43:45.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f746806bbd0 msgr2=0x7f746806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f746806bbd0 0x7f746806e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f747c034000 tx=0x7f747c007000 comp rx=0 tx=0).stop 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 msgr2=0x7f748815e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f748815e6f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7480011990 tx=0x7f7480011e50 comp rx=0 tx=0).stop 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 shutdown_connections 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f746806bbd0 0x7f746806e090 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488151c40 0x7f74880afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7488160b60 0x7f7488151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 --2- 172.21.15.6:0/3019492515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7470004800 0x7f748815e6f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 >> 172.21.15.6:0/3019492515 conn(0x7f748809ae20 msgr2=0x7f74880cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 shutdown_connections 2024-03-18T14:43:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.344+0000 7f748c9c6640 1 -- 172.21.15.6:0/3019492515 wait complete. 2024-03-18T14:43:45.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.460+0000 7f60b35c4640 1 Processor -- start 2024-03-18T14:43:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.461+0000 7f60b35c4640 1 -- start start 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.461+0000 7f60b35c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 0x7f60ac162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.461+0000 7f60b35c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60ac0b9660 con 0x7f60ac0ced00 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.461+0000 7f60b35c4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60ac0ce390 con 0x7f60ac0d3210 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.461+0000 7f60b35c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f60ac0d2270 con 0x7f60ac15e020 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60b0b5f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f60ac0ced00 0x7f60ac0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58084/0 (socket says 172.21.15.6:58084) 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60b0b5f640 1 -- 172.21.15.6:0/3359750329 learned_addr learned my addr 172.21.15.6:0/3359750329 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60b1360640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 0x7f60ac162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1482537615 0 0) 0x7f60ac0b9660 con 0x7f60ac0ced00 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6098003520 con 0x7f60ac0ced00 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 673943636 0 0) 0x7f6098003520 con 0x7f60ac0ced00 2024-03-18T14:43:45.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 msgr2=0x7f60ac162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 0x7f60ac162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 >> v1:172.21.15.92:6789/0 conn(0x7f60ac0d3210 legacy=0x7f60ac0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac0b9900 con 0x7f60ac0ced00 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f609c001670 con 0x7f60ac0ced00 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60b1360640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 0x7f60ac162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 >> v1:172.21.15.6:6789/0 conn(0x7f60ac0ced00 legacy=0x7f60ac0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60980039e0 0x7f6098003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 0x7f6098004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6098004d40 0x7f60980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac0d2270 con 0x7f6098004d40 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f609800ba70 con 0x7f60980039e0 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f609800bc40 con 0x7f60ac15e020 2024-03-18T14:43:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60b1b61640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6098004d40 0x7f60980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60b1360640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60980039e0 0x7f6098003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.462+0000 7f60b0b5f640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 0x7f6098004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.463+0000 7f60b1b61640 1 -- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 msgr2=0x7f6098004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.463+0000 7f60b1b61640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15e020 0x7f6098004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.463+0000 7f60b1b61640 1 -- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60980039e0 msgr2=0x7f6098003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.463+0000 7f60b1b61640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60980039e0 0x7f6098003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.463+0000 7f60b1b61640 1 -- 172.21.15.6:0/3359750329 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6098003520 con 0x7f6098004d40 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.463+0000 7f60b1b61640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6098004d40 0x7f60980051c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f60a4001f90 tx=0x7f6098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd0a4c278a28a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.463+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4001b30 con 0x7f6098004d40 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a402d710 con 0x7f6098004d40 2024-03-18T14:43:45.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60a3fff640 1 -- 172.21.15.6:0/3359750329 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a402db00 con 0x7f6098004d40 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60b35c4640 1 -- 172.21.15.6:0/3359750329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6098004d40 msgr2=0x7f60980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60b35c4640 1 --2- 172.21.15.6:0/3359750329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6098004d40 0x7f60980051c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f60a4001f90 tx=0x7f6098005830 comp rx=0 tx=0).stop 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60b1b61640 1 -- 172.21.15.6:0/3359750329 reap_dead start 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60b35c4640 1 -- 172.21.15.6:0/3359750329 shutdown_connections 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60b35c4640 1 -- 172.21.15.6:0/3359750329 >> 172.21.15.6:0/3359750329 conn(0x7f60ac072ce0 msgr2=0x7f60ac0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60b35c4640 1 -- 172.21.15.6:0/3359750329 shutdown_connections 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.464+0000 7f60b35c4640 1 -- 172.21.15.6:0/3359750329 wait complete. 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.465+0000 7f60b35c4640 1 Processor -- start 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.465+0000 7f60b35c4640 1 -- start start 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b35c4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6098004d40 0x7f60ac152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b35c4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60ac0b4de0 0x7f60ac15fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b35c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15abe0 0x7f60ac15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b35c4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac0cf2c0 con 0x7f60ac0b4de0 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b35c4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac0ce890 con 0x7f6098004d40 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b35c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f60ac0cf440 con 0x7f60ac15abe0 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60ac0b4de0 0x7f60ac15fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60ac0b4de0 0x7f60ac15fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36016/0 (socket says 172.21.15.6:36016) 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 -- 172.21.15.6:0/1180421584 learned_addr learned my addr 172.21.15.6:0/1180421584 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b1b61640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15abe0 0x7f60ac15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b1360640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6098004d40 0x7f60ac152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 -- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15abe0 msgr2=0x7f60ac15b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15abe0 0x7f60ac15b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 -- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6098004d40 msgr2=0x7f60ac152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6098004d40 0x7f60ac152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b0b5f640 1 -- 172.21.15.6:0/1180421584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac1603a0 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b1b61640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15abe0 0x7f60ac15b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.466+0000 7f60b1360640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6098004d40 0x7f60ac152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.467+0000 7f60b0b5f640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60ac0b4de0 0x7f60ac15fa30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f609c0072f0 tx=0x7f609c0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.467+0000 7f60a1ffb640 1 -- 172.21.15.6:0/1180421584 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c00f960 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.467+0000 7f60a1ffb640 1 -- 172.21.15.6:0/1180421584 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c018070 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.467+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ac160680 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.467+0000 7f60a1ffb640 1 -- 172.21.15.6:0/1180421584 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c00fd60 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.467+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ac160c40 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.468+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ac08afe0 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.469+0000 7f60a1ffb640 1 -- 172.21.15.6:0/1180421584 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f609c01d460 con 0x7f60ac0b4de0 2024-03-18T14:43:45.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.469+0000 7f60a1ffb640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f609006bbd0 0x7f609006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.469+0000 7f60a1ffb640 1 -- 172.21.15.6:0/1180421584 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f609c08adb0 con 0x7f60ac0b4de0 2024-03-18T14:43:45.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.470+0000 7f60b1360640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f609006bbd0 0x7f609006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.470+0000 7f60b1360640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f609006bbd0 0x7f609006e090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009ab0 tx=0x7f60a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.474+0000 7f60a1ffb640 1 -- 172.21.15.6:0/1180421584 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f609c08f070 con 0x7f60ac0b4de0 2024-03-18T14:43:45.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.657+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60ac0d26a0 con 0x7f60ac0b4de0 2024-03-18T14:43:45.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.658+0000 7f60a1ffb640 1 -- 172.21.15.6:0/1180421584 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7f609c0595e0 con 0x7f60ac0b4de0 2024-03-18T14:43:45.660 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:45.660 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f609006bbd0 msgr2=0x7f609006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f609006bbd0 0x7f609006e090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009ab0 tx=0x7f60a8008040 comp rx=0 tx=0).stop 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60ac0b4de0 msgr2=0x7f60ac15fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60ac0b4de0 0x7f60ac15fa30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f609c0072f0 tx=0x7f609c0077b0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 shutdown_connections 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f609006bbd0 0x7f609006e090 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60ac15abe0 0x7f60ac15b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60ac0b4de0 0x7f60ac15fa30 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 --2- 172.21.15.6:0/1180421584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6098004d40 0x7f60ac152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 >> 172.21.15.6:0/1180421584 conn(0x7f60ac072ce0 msgr2=0x7f60ac152960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 shutdown_connections 2024-03-18T14:43:45.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.660+0000 7f60b35c4640 1 -- 172.21.15.6:0/1180421584 wait complete. 2024-03-18T14:43:45.673 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T14:43:45.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:43:45.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.773+0000 7ff843e85640 1 Processor -- start 2024-03-18T14:43:45.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.773+0000 7ff843e85640 1 -- start start 2024-03-18T14:43:45.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff843e85640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff843e85640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff83c161a30 0x7ff83c0ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff843e85640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83c0d2100 con 0x7ff83c0b9650 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff843e85640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff83c15e3b0 con 0x7ff83c161a30 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff843e85640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff83c161f60 con 0x7ff83c15dfd0 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841420640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff83c161a30 0x7ff83c0ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841c21640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841c21640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c0ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55276/0 (socket says 172.21.15.6:55276) 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841420640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff83c161a30 0x7ff83c0ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46744/0 (socket says 172.21.15.6:46744) 2024-03-18T14:43:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841c21640 1 -- 172.21.15.6:0/153676722 learned_addr learned my addr 172.21.15.6:0/153676722 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:45.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841c21640 1 -- 172.21.15.6:0/153676722 >> v2:172.21.15.92:3300/0 conn(0x7ff83c161a30 msgr2=0x7ff83c0ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:45.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841c21640 1 --2- 172.21.15.6:0/153676722 >> v2:172.21.15.92:3300/0 conn(0x7ff83c161a30 0x7ff83c0ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841c21640 1 -- 172.21.15.6:0/153676722 >> v1:172.21.15.92:6789/0 conn(0x7ff83c0b9650 legacy=0x7ff83c0b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841c21640 1 -- 172.21.15.6:0/153676722 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83c162240 con 0x7ff83c15dfd0 2024-03-18T14:43:45.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.774+0000 7ff841420640 1 --2- 172.21.15.6:0/153676722 >> v2:172.21.15.92:3300/0 conn(0x7ff83c161a30 0x7ff83c0ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff841c21640 1 --2- 172.21.15.6:0/153676722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c0ce990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff838009870 tx=0x7ff83802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f830ea19c5f2ca3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff840c1f640 1 -- 172.21.15.6:0/153676722 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838034070 con 0x7ff83c15dfd0 2024-03-18T14:43:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff840c1f640 1 -- 172.21.15.6:0/153676722 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff838030840 con 0x7ff83c15dfd0 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff840c1f640 1 -- 172.21.15.6:0/153676722 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838030c10 con 0x7ff83c15dfd0 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 -- 172.21.15.6:0/153676722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 msgr2=0x7ff83c0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 --2- 172.21.15.6:0/153676722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c0ce990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff838009870 tx=0x7ff83802b980 comp rx=0 tx=0).stop 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 -- 172.21.15.6:0/153676722 shutdown_connections 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 --2- 172.21.15.6:0/153676722 >> v2:172.21.15.92:3300/0 conn(0x7ff83c161a30 0x7ff83c0ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 --2- 172.21.15.6:0/153676722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c0ce990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 -- 172.21.15.6:0/153676722 >> 172.21.15.6:0/153676722 conn(0x7ff83c06c850 msgr2=0x7ff83c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 -- 172.21.15.6:0/153676722 shutdown_connections 2024-03-18T14:43:45.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.776+0000 7ff843e85640 1 -- 172.21.15.6:0/153676722 wait complete. 2024-03-18T14:43:45.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.777+0000 7ff843e85640 1 Processor -- start 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff843e85640 1 -- start start 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff843e85640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff83c0b9650 0x7ff83c0cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff843e85640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff843e85640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83c161a30 0x7ff83c0b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff843e85640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff83c0d1030 con 0x7ff83c0b9650 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff843e85640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff83c162800 con 0x7ff83c161a30 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff843e85640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff83c0d11b0 con 0x7ff83c15dfd0 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff841c21640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff83c0b9650 0x7ff83c0cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff841c21640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff83c0b9650 0x7ff83c0cb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36030/0 (socket says 172.21.15.6:36030) 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.778+0000 7ff841c21640 1 -- 172.21.15.6:0/992592807 learned_addr learned my addr 172.21.15.6:0/992592807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff841420640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff842422640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83c161a30 0x7ff83c0b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff841c21640 1 -- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 msgr2=0x7ff83c160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff841c21640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff841c21640 1 -- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83c161a30 msgr2=0x7ff83c0b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff841c21640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83c161a30 0x7ff83c0b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff841c21640 1 -- 172.21.15.6:0/992592807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83c0ccad0 con 0x7ff83c0b9650 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff841c21640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff83c0b9650 0x7ff83c0cb770 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff83802b9c0 tx=0x7ff838004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff8327fc640 1 -- 172.21.15.6:0/992592807 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838048070 con 0x7ff83c0b9650 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff8327fc640 1 -- 172.21.15.6:0/992592807 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff838030640 con 0x7ff83c0b9650 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83c0ccdb0 con 0x7ff83c0b9650 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff8327fc640 1 -- 172.21.15.6:0/992592807 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838034040 con 0x7ff83c0b9650 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.779+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff83c0cd370 con 0x7ff83c0b9650 2024-03-18T14:43:45.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.780+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff83c0cd6a0 con 0x7ff83c0b9650 2024-03-18T14:43:45.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.781+0000 7ff8327fc640 1 -- 172.21.15.6:0/992592807 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff838037070 con 0x7ff83c0b9650 2024-03-18T14:43:45.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.781+0000 7ff8327fc640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff82006bbd0 0x7ff82006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:45.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.781+0000 7ff8327fc640 1 -- 172.21.15.6:0/992592807 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7ff8380ac750 con 0x7ff83c0b9650 2024-03-18T14:43:45.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.782+0000 7ff841420640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff82006bbd0 0x7ff82006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:45.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.782+0000 7ff841420640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff82006bbd0 0x7ff82006e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009560 tx=0x7ff82c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:45.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.787+0000 7ff8327fc640 1 -- 172.21.15.6:0/992592807 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff838030810 con 0x7ff83c0b9650 2024-03-18T14:43:45.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.965+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff83c04f620 con 0x7ff82006bbd0 2024-03-18T14:43:45.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.972+0000 7ff8327fc640 1 -- 172.21.15.6:0/992592807 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251954 (secure 0 0 0) 0x7ff83c04f620 con 0x7ff82006bbd0 2024-03-18T14:43:45.976 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:45.978 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.974+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff82006bbd0 msgr2=0x7ff82006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.974+0000 7ff843e85640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff82006bbd0 0x7ff82006e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009560 tx=0x7ff82c009190 comp rx=0 tx=0).stop 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.974+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff83c0b9650 msgr2=0x7ff83c0cb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.974+0000 7ff843e85640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff83c0b9650 0x7ff83c0cb770 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff83802b9c0 tx=0x7ff838004730 comp rx=0 tx=0).stop 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 shutdown_connections 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff82006bbd0 0x7ff82006e090 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83c161a30 0x7ff83c0b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83c15dfd0 0x7ff83c160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 --2- 172.21.15.6:0/992592807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff83c0b9650 0x7ff83c0cb770 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:45.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 >> 172.21.15.6:0/992592807 conn(0x7ff83c06c850 msgr2=0x7ff83c0d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:45.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 shutdown_connections 2024-03-18T14:43:45.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:45.975+0000 7ff843e85640 1 -- 172.21.15.6:0/992592807 wait complete. 2024-03-18T14:43:45.988 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-03-18T14:43:45.408569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222604,"kb_used_data":6424,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752116,"statfs":{"total":773094113280,"available":772866166784,"internally_reserved":0,"allocated":6578176,"data_stored":3591980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.448003"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974008+0000","last_change":"2024-03-18T14:43:41.978141+0000","last_active":"2024-03-18T14:43:42.974008+0000","last_peered":"2024-03-18T14:43:42.974008+0000","last_clean":"2024-03-18T14:43:42.974008+0000","last_became_active":"2024-03-18T14:43:41.977839+0000","last_became_peered":"2024-03-18T14:43:41.977839+0000","last_unstale":"2024-03-18T14:43:42.974008+0000","last_undegraded":"2024-03-18T14:43:42.974008+0000","last_fullsized":"2024-03-18T14:43:42.974008+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:49:20.830857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973743+0000","last_change":"2024-03-18T14:43:41.976097+0000","last_active":"2024-03-18T14:43:42.973743+0000","last_peered":"2024-03-18T14:43:42.973743+0000","last_clean":"2024-03-18T14:43:42.973743+0000","last_became_active":"2024-03-18T14:43:41.975825+0000","last_became_peered":"2024-03-18T14:43:41.975825+0000","last_unstale":"2024-03-18T14:43:42.973743+0000","last_undegraded":"2024-03-18T14:43:42.973743+0000","last_fullsized":"2024-03-18T14:43:42.973743+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:45:33.774134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974306+0000","last_change":"2024-03-18T14:43:41.988970+0000","last_active":"2024-03-18T14:43:42.974306+0000","last_peered":"2024-03-18T14:43:42.974306+0000","last_clean":"2024-03-18T14:43:42.974306+0000","last_became_active":"2024-03-18T14:43:41.988696+0000","last_became_peered":"2024-03-18T14:43:41.988696+0000","last_unstale":"2024-03-18T14:43:42.974306+0000","last_undegraded":"2024-03-18T14:43:42.974306+0000","last_fullsized":"2024-03-18T14:43:42.974306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:56.329557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974477+0000","last_change":"2024-03-18T14:43:41.989221+0000","last_active":"2024-03-18T14:43:42.974477+0000","last_peered":"2024-03-18T14:43:42.974477+0000","last_clean":"2024-03-18T14:43:42.974477+0000","last_became_active":"2024-03-18T14:43:41.987976+0000","last_became_peered":"2024-03-18T14:43:41.987976+0000","last_unstale":"2024-03-18T14:43:42.974477+0000","last_undegraded":"2024-03-18T14:43:42.974477+0000","last_fullsized":"2024-03-18T14:43:42.974477+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:08.711617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973558+0000","last_change":"2024-03-18T14:43:41.987473+0000","last_active":"2024-03-18T14:43:42.973558+0000","last_peered":"2024-03-18T14:43:42.973558+0000","last_clean":"2024-03-18T14:43:42.973558+0000","last_became_active":"2024-03-18T14:43:41.987208+0000","last_became_peered":"2024-03-18T14:43:41.987208+0000","last_unstale":"2024-03-18T14:43:42.973558+0000","last_undegraded":"2024-03-18T14:43:42.973558+0000","last_fullsized":"2024-03-18T14:43:42.973558+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:47.711875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971160+0000","last_change":"2024-03-18T14:43:41.994240+0000","last_active":"2024-03-18T14:43:42.971160+0000","last_peered":"2024-03-18T14:43:42.971160+0000","last_clean":"2024-03-18T14:43:42.971160+0000","last_became_active":"2024-03-18T14:43:41.993537+0000","last_became_peered":"2024-03-18T14:43:41.993537+0000","last_unstale":"2024-03-18T14:43:42.971160+0000","last_undegraded":"2024-03-18T14:43:42.971160+0000","last_fullsized":"2024-03-18T14:43:42.971160+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:15.827363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974992+0000","last_change":"2024-03-18T14:43:41.988101+0000","last_active":"2024-03-18T14:43:42.974992+0000","last_peered":"2024-03-18T14:43:42.974992+0000","last_clean":"2024-03-18T14:43:42.974992+0000","last_became_active":"2024-03-18T14:43:41.987819+0000","last_became_peered":"2024-03-18T14:43:41.987819+0000","last_unstale":"2024-03-18T14:43:42.974992+0000","last_undegraded":"2024-03-18T14:43:42.974992+0000","last_fullsized":"2024-03-18T14:43:42.974992+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:21.618875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973298+0000","last_change":"2024-03-18T14:43:41.981904+0000","last_active":"2024-03-18T14:43:42.973298+0000","last_peered":"2024-03-18T14:43:42.973298+0000","last_clean":"2024-03-18T14:43:42.973298+0000","last_became_active":"2024-03-18T14:43:41.980691+0000","last_became_peered":"2024-03-18T14:43:41.980691+0000","last_unstale":"2024-03-18T14:43:42.973298+0000","last_undegraded":"2024-03-18T14:43:42.973298+0000","last_fullsized":"2024-03-18T14:43:42.973298+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:17:28.166306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975085+0000","last_change":"2024-03-18T14:43:41.989662+0000","last_active":"2024-03-18T14:43:42.975085+0000","last_peered":"2024-03-18T14:43:42.975085+0000","last_clean":"2024-03-18T14:43:42.975085+0000","last_became_active":"2024-03-18T14:43:41.989148+0000","last_became_peered":"2024-03-18T14:43:41.989148+0000","last_unstale":"2024-03-18T14:43:42.975085+0000","last_undegraded":"2024-03-18T14:43:42.975085+0000","last_fullsized":"2024-03-18T14:43:42.975085+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:20:47.340218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970250+0000","last_change":"2024-03-18T14:43:41.987080+0000","last_active":"2024-03-18T14:43:42.970250+0000","last_peered":"2024-03-18T14:43:42.970250+0000","last_clean":"2024-03-18T14:43:42.970250+0000","last_became_active":"2024-03-18T14:43:41.986789+0000","last_became_peered":"2024-03-18T14:43:41.986789+0000","last_unstale":"2024-03-18T14:43:42.970250+0000","last_undegraded":"2024-03-18T14:43:42.970250+0000","last_fullsized":"2024-03-18T14:43:42.970250+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:49.721963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970573+0000","last_change":"2024-03-18T14:43:41.976980+0000","last_active":"2024-03-18T14:43:42.970573+0000","last_peered":"2024-03-18T14:43:42.970573+0000","last_clean":"2024-03-18T14:43:42.970573+0000","last_became_active":"2024-03-18T14:43:41.976611+0000","last_became_peered":"2024-03-18T14:43:41.976611+0000","last_unstale":"2024-03-18T14:43:42.970573+0000","last_undegraded":"2024-03-18T14:43:42.970573+0000","last_fullsized":"2024-03-18T14:43:42.970573+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:28:30.289262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974494+0000","last_change":"2024-03-18T14:43:41.990208+0000","last_active":"2024-03-18T14:43:42.974494+0000","last_peered":"2024-03-18T14:43:42.974494+0000","last_clean":"2024-03-18T14:43:42.974494+0000","last_became_active":"2024-03-18T14:43:41.989913+0000","last_became_peered":"2024-03-18T14:43:41.989913+0000","last_unstale":"2024-03-18T14:43:42.974494+0000","last_undegraded":"2024-03-18T14:43:42.974494+0000","last_fullsized":"2024-03-18T14:43:42.974494+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:22.376659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971540+0000","last_change":"2024-03-18T14:43:41.991068+0000","last_active":"2024-03-18T14:43:42.971540+0000","last_peered":"2024-03-18T14:43:42.971540+0000","last_clean":"2024-03-18T14:43:42.971540+0000","last_became_active":"2024-03-18T14:43:41.990801+0000","last_became_peered":"2024-03-18T14:43:41.990801+0000","last_unstale":"2024-03-18T14:43:42.971540+0000","last_undegraded":"2024-03-18T14:43:42.971540+0000","last_fullsized":"2024-03-18T14:43:42.971540+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.004568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975089+0000","last_change":"2024-03-18T14:43:41.987107+0000","last_active":"2024-03-18T14:43:42.975089+0000","last_peered":"2024-03-18T14:43:42.975089+0000","last_clean":"2024-03-18T14:43:42.975089+0000","last_became_active":"2024-03-18T14:43:41.986829+0000","last_became_peered":"2024-03-18T14:43:41.986829+0000","last_unstale":"2024-03-18T14:43:42.975089+0000","last_undegraded":"2024-03-18T14:43:42.975089+0000","last_fullsized":"2024-03-18T14:43:42.975089+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:31.332178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974995+0000","last_change":"2024-03-18T14:43:41.992096+0000","last_active":"2024-03-18T14:43:42.974995+0000","last_peered":"2024-03-18T14:43:42.974995+0000","last_clean":"2024-03-18T14:43:42.974995+0000","last_became_active":"2024-03-18T14:43:41.991792+0000","last_became_peered":"2024-03-18T14:43:41.991792+0000","last_unstale":"2024-03-18T14:43:42.974995+0000","last_undegraded":"2024-03-18T14:43:42.974995+0000","last_fullsized":"2024-03-18T14:43:42.974995+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:00.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970872+0000","last_change":"2024-03-18T14:43:41.996226+0000","last_active":"2024-03-18T14:43:42.970872+0000","last_peered":"2024-03-18T14:43:42.970872+0000","last_clean":"2024-03-18T14:43:42.970872+0000","last_became_active":"2024-03-18T14:43:41.995906+0000","last_became_peered":"2024-03-18T14:43:41.995906+0000","last_unstale":"2024-03-18T14:43:42.970872+0000","last_undegraded":"2024-03-18T14:43:42.970872+0000","last_fullsized":"2024-03-18T14:43:42.970872+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:39.379173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974000+0000","last_change":"2024-03-18T14:43:41.991239+0000","last_active":"2024-03-18T14:43:42.974000+0000","last_peered":"2024-03-18T14:43:42.974000+0000","last_clean":"2024-03-18T14:43:42.974000+0000","last_became_active":"2024-03-18T14:43:41.990929+0000","last_became_peered":"2024-03-18T14:43:41.990929+0000","last_unstale":"2024-03-18T14:43:42.974000+0000","last_undegraded":"2024-03-18T14:43:42.974000+0000","last_fullsized":"2024-03-18T14:43:42.974000+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:44:34.079834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974248+0000","last_change":"2024-03-18T14:43:41.999148+0000","last_active":"2024-03-18T14:43:42.974248+0000","last_peered":"2024-03-18T14:43:42.974248+0000","last_clean":"2024-03-18T14:43:42.974248+0000","last_became_active":"2024-03-18T14:43:41.998903+0000","last_became_peered":"2024-03-18T14:43:41.998903+0000","last_unstale":"2024-03-18T14:43:42.974248+0000","last_undegraded":"2024-03-18T14:43:42.974248+0000","last_fullsized":"2024-03-18T14:43:42.974248+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:50.325632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973589+0000","last_change":"2024-03-18T14:43:41.978655+0000","last_active":"2024-03-18T14:43:42.973589+0000","last_peered":"2024-03-18T14:43:42.973589+0000","last_clean":"2024-03-18T14:43:42.973589+0000","last_became_active":"2024-03-18T14:43:41.977381+0000","last_became_peered":"2024-03-18T14:43:41.977381+0000","last_unstale":"2024-03-18T14:43:42.973589+0000","last_undegraded":"2024-03-18T14:43:42.973589+0000","last_fullsized":"2024-03-18T14:43:42.973589+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:06.678808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973610+0000","last_change":"2024-03-18T14:43:41.988740+0000","last_active":"2024-03-18T14:43:42.973610+0000","last_peered":"2024-03-18T14:43:42.973610+0000","last_clean":"2024-03-18T14:43:42.973610+0000","last_became_active":"2024-03-18T14:43:41.988378+0000","last_became_peered":"2024-03-18T14:43:41.988378+0000","last_unstale":"2024-03-18T14:43:42.973610+0000","last_undegraded":"2024-03-18T14:43:42.973610+0000","last_fullsized":"2024-03-18T14:43:42.973610+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:02:47.517991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973825+0000","last_change":"2024-03-18T14:43:41.989206+0000","last_active":"2024-03-18T14:43:42.973825+0000","last_peered":"2024-03-18T14:43:42.973825+0000","last_clean":"2024-03-18T14:43:42.973825+0000","last_became_active":"2024-03-18T14:43:41.988919+0000","last_became_peered":"2024-03-18T14:43:41.988919+0000","last_unstale":"2024-03-18T14:43:42.973825+0000","last_undegraded":"2024-03-18T14:43:42.973825+0000","last_fullsized":"2024-03-18T14:43:42.973825+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:49:52.103790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973982+0000","last_change":"2024-03-18T14:43:41.976324+0000","last_active":"2024-03-18T14:43:42.973982+0000","last_peered":"2024-03-18T14:43:42.973982+0000","last_clean":"2024-03-18T14:43:42.973982+0000","last_became_active":"2024-03-18T14:43:41.975861+0000","last_became_peered":"2024-03-18T14:43:41.975861+0000","last_unstale":"2024-03-18T14:43:42.973982+0000","last_undegraded":"2024-03-18T14:43:42.973982+0000","last_fullsized":"2024-03-18T14:43:42.973982+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:16.750519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973817+0000","last_change":"2024-03-18T14:43:41.987768+0000","last_active":"2024-03-18T14:43:42.973817+0000","last_peered":"2024-03-18T14:43:42.973817+0000","last_clean":"2024-03-18T14:43:42.973817+0000","last_became_active":"2024-03-18T14:43:41.987494+0000","last_became_peered":"2024-03-18T14:43:41.987494+0000","last_unstale":"2024-03-18T14:43:42.973817+0000","last_undegraded":"2024-03-18T14:43:42.973817+0000","last_fullsized":"2024-03-18T14:43:42.973817+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:34.499335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971158+0000","last_change":"2024-03-18T14:43:41.998908+0000","last_active":"2024-03-18T14:43:42.971158+0000","last_peered":"2024-03-18T14:43:42.971158+0000","last_clean":"2024-03-18T14:43:42.971158+0000","last_became_active":"2024-03-18T14:43:41.998510+0000","last_became_peered":"2024-03-18T14:43:41.998510+0000","last_unstale":"2024-03-18T14:43:42.971158+0000","last_undegraded":"2024-03-18T14:43:42.971158+0000","last_fullsized":"2024-03-18T14:43:42.971158+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:20.750238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974151+0000","last_change":"2024-03-18T14:43:41.977490+0000","last_active":"2024-03-18T14:43:42.974151+0000","last_peered":"2024-03-18T14:43:42.974151+0000","last_clean":"2024-03-18T14:43:42.974151+0000","last_became_active":"2024-03-18T14:43:41.977174+0000","last_became_peered":"2024-03-18T14:43:41.977174+0000","last_unstale":"2024-03-18T14:43:42.974151+0000","last_undegraded":"2024-03-18T14:43:42.974151+0000","last_fullsized":"2024-03-18T14:43:42.974151+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:33:19.352003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973721+0000","last_change":"2024-03-18T14:43:41.994593+0000","last_active":"2024-03-18T14:43:42.973721+0000","last_peered":"2024-03-18T14:43:42.973721+0000","last_clean":"2024-03-18T14:43:42.973721+0000","last_became_active":"2024-03-18T14:43:41.993184+0000","last_became_peered":"2024-03-18T14:43:41.993184+0000","last_unstale":"2024-03-18T14:43:42.973721+0000","last_undegraded":"2024-03-18T14:43:42.973721+0000","last_fullsized":"2024-03-18T14:43:42.973721+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:50.325632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971163+0000","last_change":"2024-03-18T14:43:41.994739+0000","last_active":"2024-03-18T14:43:42.971163+0000","last_peered":"2024-03-18T14:43:42.971163+0000","last_clean":"2024-03-18T14:43:42.971163+0000","last_became_active":"2024-03-18T14:43:41.994456+0000","last_became_peered":"2024-03-18T14:43:41.994456+0000","last_unstale":"2024-03-18T14:43:42.971163+0000","last_undegraded":"2024-03-18T14:43:42.971163+0000","last_fullsized":"2024-03-18T14:43:42.971163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:50.325632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973854+0000","last_change":"2024-03-18T14:43:41.986786+0000","last_active":"2024-03-18T14:43:42.973854+0000","last_peered":"2024-03-18T14:43:42.973854+0000","last_clean":"2024-03-18T14:43:42.973854+0000","last_became_active":"2024-03-18T14:43:41.986510+0000","last_became_peered":"2024-03-18T14:43:41.986510+0000","last_unstale":"2024-03-18T14:43:42.973854+0000","last_undegraded":"2024-03-18T14:43:42.973854+0000","last_fullsized":"2024-03-18T14:43:42.973854+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:02.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973784+0000","last_change":"2024-03-18T14:43:41.987134+0000","last_active":"2024-03-18T14:43:42.973784+0000","last_peered":"2024-03-18T14:43:42.973784+0000","last_clean":"2024-03-18T14:43:42.973784+0000","last_became_active":"2024-03-18T14:43:41.986850+0000","last_became_peered":"2024-03-18T14:43:41.986850+0000","last_unstale":"2024-03-18T14:43:42.973784+0000","last_undegraded":"2024-03-18T14:43:42.973784+0000","last_fullsized":"2024-03-18T14:43:42.973784+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:41.388142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974057+0000","last_change":"2024-03-18T14:43:41.982458+0000","last_active":"2024-03-18T14:43:42.974057+0000","last_peered":"2024-03-18T14:43:42.974057+0000","last_clean":"2024-03-18T14:43:42.974057+0000","last_became_active":"2024-03-18T14:43:41.982026+0000","last_became_peered":"2024-03-18T14:43:41.982026+0000","last_unstale":"2024-03-18T14:43:42.974057+0000","last_undegraded":"2024-03-18T14:43:42.974057+0000","last_fullsized":"2024-03-18T14:43:42.974057+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:46:10.236105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973880+0000","last_change":"2024-03-18T14:43:41.975787+0000","last_active":"2024-03-18T14:43:42.973880+0000","last_peered":"2024-03-18T14:43:42.973880+0000","last_clean":"2024-03-18T14:43:42.973880+0000","last_became_active":"2024-03-18T14:43:41.975477+0000","last_became_peered":"2024-03-18T14:43:41.975477+0000","last_unstale":"2024-03-18T14:43:42.973880+0000","last_undegraded":"2024-03-18T14:43:42.973880+0000","last_fullsized":"2024-03-18T14:43:42.973880+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:56.155609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970909+0000","last_change":"2024-03-18T14:43:42.000064+0000","last_active":"2024-03-18T14:43:42.970909+0000","last_peered":"2024-03-18T14:43:42.970909+0000","last_clean":"2024-03-18T14:43:42.970909+0000","last_became_active":"2024-03-18T14:43:41.999677+0000","last_became_peered":"2024-03-18T14:43:41.999677+0000","last_unstale":"2024-03-18T14:43:42.970909+0000","last_undegraded":"2024-03-18T14:43:42.970909+0000","last_fullsized":"2024-03-18T14:43:42.970909+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:38:56.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971875+0000","last_change":"2024-03-18T14:43:41.990616+0000","last_active":"2024-03-18T14:43:42.971875+0000","last_peered":"2024-03-18T14:43:42.971875+0000","last_clean":"2024-03-18T14:43:42.971875+0000","last_became_active":"2024-03-18T14:43:41.990303+0000","last_became_peered":"2024-03-18T14:43:41.990303+0000","last_unstale":"2024-03-18T14:43:42.971875+0000","last_undegraded":"2024-03-18T14:43:42.971875+0000","last_fullsized":"2024-03-18T14:43:42.971875+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:05.007512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971786+0000","last_change":"2024-03-18T14:43:40.983800+0000","last_active":"2024-03-18T14:43:42.971786+0000","last_peered":"2024-03-18T14:43:42.971786+0000","last_clean":"2024-03-18T14:43:42.971786+0000","last_became_active":"2024-03-18T14:43:40.983512+0000","last_became_peered":"2024-03-18T14:43:40.983512+0000","last_unstale":"2024-03-18T14:43:42.971786+0000","last_undegraded":"2024-03-18T14:43:42.971786+0000","last_fullsized":"2024-03-18T14:43:42.971786+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:39:45.521393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970408+0000","last_change":"2024-03-18T14:43:41.978941+0000","last_active":"2024-03-18T14:43:42.970408+0000","last_peered":"2024-03-18T14:43:42.970408+0000","last_clean":"2024-03-18T14:43:42.970408+0000","last_became_active":"2024-03-18T14:43:41.978681+0000","last_became_peered":"2024-03-18T14:43:41.978681+0000","last_unstale":"2024-03-18T14:43:42.970408+0000","last_undegraded":"2024-03-18T14:43:42.970408+0000","last_fullsized":"2024-03-18T14:43:42.970408+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:46:19.054657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973825+0000","last_change":"2024-03-18T14:43:40.986288+0000","last_active":"2024-03-18T14:43:42.973825+0000","last_peered":"2024-03-18T14:43:42.973825+0000","last_clean":"2024-03-18T14:43:42.973825+0000","last_became_active":"2024-03-18T14:43:40.985971+0000","last_became_peered":"2024-03-18T14:43:40.985971+0000","last_unstale":"2024-03-18T14:43:42.973825+0000","last_undegraded":"2024-03-18T14:43:42.973825+0000","last_fullsized":"2024-03-18T14:43:42.973825+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:00.912885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971479+0000","last_change":"2024-03-18T14:43:41.995874+0000","last_active":"2024-03-18T14:43:42.971479+0000","last_peered":"2024-03-18T14:43:42.971479+0000","last_clean":"2024-03-18T14:43:42.971479+0000","last_became_active":"2024-03-18T14:43:41.994701+0000","last_became_peered":"2024-03-18T14:43:41.994701+0000","last_unstale":"2024-03-18T14:43:42.971479+0000","last_undegraded":"2024-03-18T14:43:42.971479+0000","last_fullsized":"2024-03-18T14:43:42.971479+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:08.711617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971564+0000","last_change":"2024-03-18T14:43:40.983218+0000","last_active":"2024-03-18T14:43:42.971564+0000","last_peered":"2024-03-18T14:43:42.971564+0000","last_clean":"2024-03-18T14:43:42.971564+0000","last_became_active":"2024-03-18T14:43:40.982929+0000","last_became_peered":"2024-03-18T14:43:40.982929+0000","last_unstale":"2024-03-18T14:43:42.971564+0000","last_undegraded":"2024-03-18T14:43:42.971564+0000","last_fullsized":"2024-03-18T14:43:42.971564+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:30.323270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971301+0000","last_change":"2024-03-18T14:43:41.993827+0000","last_active":"2024-03-18T14:43:42.971301+0000","last_peered":"2024-03-18T14:43:42.971301+0000","last_clean":"2024-03-18T14:43:42.971301+0000","last_became_active":"2024-03-18T14:43:41.993460+0000","last_became_peered":"2024-03-18T14:43:41.993460+0000","last_unstale":"2024-03-18T14:43:42.971301+0000","last_undegraded":"2024-03-18T14:43:42.971301+0000","last_fullsized":"2024-03-18T14:43:42.971301+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:56.329557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974587+0000","last_change":"2024-03-18T14:43:40.986774+0000","last_active":"2024-03-18T14:43:42.974587+0000","last_peered":"2024-03-18T14:43:42.974587+0000","last_clean":"2024-03-18T14:43:42.974587+0000","last_became_active":"2024-03-18T14:43:40.986438+0000","last_became_peered":"2024-03-18T14:43:40.986438+0000","last_unstale":"2024-03-18T14:43:42.974587+0000","last_undegraded":"2024-03-18T14:43:42.974587+0000","last_fullsized":"2024-03-18T14:43:42.974587+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:58.935947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973475+0000","last_change":"2024-03-18T14:43:41.976102+0000","last_active":"2024-03-18T14:43:42.973475+0000","last_peered":"2024-03-18T14:43:42.973475+0000","last_clean":"2024-03-18T14:43:42.973475+0000","last_became_active":"2024-03-18T14:43:41.975779+0000","last_became_peered":"2024-03-18T14:43:41.975779+0000","last_unstale":"2024-03-18T14:43:42.973475+0000","last_undegraded":"2024-03-18T14:43:42.973475+0000","last_fullsized":"2024-03-18T14:43:42.973475+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:13.535288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974765+0000","last_change":"2024-03-18T14:43:40.986248+0000","last_active":"2024-03-18T14:43:42.974765+0000","last_peered":"2024-03-18T14:43:42.974765+0000","last_clean":"2024-03-18T14:43:42.974765+0000","last_became_active":"2024-03-18T14:43:40.985994+0000","last_became_peered":"2024-03-18T14:43:40.985994+0000","last_unstale":"2024-03-18T14:43:42.974765+0000","last_undegraded":"2024-03-18T14:43:42.974765+0000","last_fullsized":"2024-03-18T14:43:42.974765+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:50.895017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973530+0000","last_change":"2024-03-18T14:43:41.990665+0000","last_active":"2024-03-18T14:43:42.973530+0000","last_peered":"2024-03-18T14:43:42.973530+0000","last_clean":"2024-03-18T14:43:42.973530+0000","last_became_active":"2024-03-18T14:43:41.990346+0000","last_became_peered":"2024-03-18T14:43:41.990346+0000","last_unstale":"2024-03-18T14:43:42.973530+0000","last_undegraded":"2024-03-18T14:43:42.973530+0000","last_fullsized":"2024-03-18T14:43:42.973530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:18.541018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971144+0000","last_change":"2024-03-18T14:43:40.983138+0000","last_active":"2024-03-18T14:43:42.971144+0000","last_peered":"2024-03-18T14:43:42.971144+0000","last_clean":"2024-03-18T14:43:42.971144+0000","last_became_active":"2024-03-18T14:43:40.982801+0000","last_became_peered":"2024-03-18T14:43:40.982801+0000","last_unstale":"2024-03-18T14:43:42.971144+0000","last_undegraded":"2024-03-18T14:43:42.971144+0000","last_fullsized":"2024-03-18T14:43:42.971144+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:20.609968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971359+0000","last_change":"2024-03-18T14:43:41.990846+0000","last_active":"2024-03-18T14:43:42.971359+0000","last_peered":"2024-03-18T14:43:42.971359+0000","last_clean":"2024-03-18T14:43:42.971359+0000","last_became_active":"2024-03-18T14:43:41.990552+0000","last_became_peered":"2024-03-18T14:43:41.990552+0000","last_unstale":"2024-03-18T14:43:42.971359+0000","last_undegraded":"2024-03-18T14:43:42.971359+0000","last_fullsized":"2024-03-18T14:43:42.971359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:33.841740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973702+0000","last_change":"2024-03-18T14:43:40.979927+0000","last_active":"2024-03-18T14:43:42.973702+0000","last_peered":"2024-03-18T14:43:42.973702+0000","last_clean":"2024-03-18T14:43:42.973702+0000","last_became_active":"2024-03-18T14:43:40.979561+0000","last_became_peered":"2024-03-18T14:43:40.979561+0000","last_unstale":"2024-03-18T14:43:42.973702+0000","last_undegraded":"2024-03-18T14:43:42.973702+0000","last_fullsized":"2024-03-18T14:43:42.973702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:26.204701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973360+0000","last_change":"2024-03-18T14:43:41.989410+0000","last_active":"2024-03-18T14:43:42.973360+0000","last_peered":"2024-03-18T14:43:42.973360+0000","last_clean":"2024-03-18T14:43:42.973360+0000","last_became_active":"2024-03-18T14:43:41.988707+0000","last_became_peered":"2024-03-18T14:43:41.988707+0000","last_unstale":"2024-03-18T14:43:42.973360+0000","last_undegraded":"2024-03-18T14:43:42.973360+0000","last_fullsized":"2024-03-18T14:43:42.973360+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:41.135260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974745+0000","last_change":"2024-03-18T14:43:40.988370+0000","last_active":"2024-03-18T14:43:42.974745+0000","last_peered":"2024-03-18T14:43:42.974745+0000","last_clean":"2024-03-18T14:43:42.974745+0000","last_became_active":"2024-03-18T14:43:40.988143+0000","last_became_peered":"2024-03-18T14:43:40.988143+0000","last_unstale":"2024-03-18T14:43:42.974745+0000","last_undegraded":"2024-03-18T14:43:42.974745+0000","last_fullsized":"2024-03-18T14:43:42.974745+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:19.741331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975115+0000","last_change":"2024-03-18T14:43:41.992354+0000","last_active":"2024-03-18T14:43:42.975115+0000","last_peered":"2024-03-18T14:43:42.975115+0000","last_clean":"2024-03-18T14:43:42.975115+0000","last_became_active":"2024-03-18T14:43:41.991543+0000","last_became_peered":"2024-03-18T14:43:41.991543+0000","last_unstale":"2024-03-18T14:43:42.975115+0000","last_undegraded":"2024-03-18T14:43:42.975115+0000","last_fullsized":"2024-03-18T14:43:42.975115+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:21.550317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974276+0000","last_change":"2024-03-18T14:43:40.990848+0000","last_active":"2024-03-18T14:43:42.974276+0000","last_peered":"2024-03-18T14:43:42.974276+0000","last_clean":"2024-03-18T14:43:42.974276+0000","last_became_active":"2024-03-18T14:43:40.990506+0000","last_became_peered":"2024-03-18T14:43:40.990506+0000","last_unstale":"2024-03-18T14:43:42.974276+0000","last_undegraded":"2024-03-18T14:43:42.974276+0000","last_fullsized":"2024-03-18T14:43:42.974276+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:18:23.568861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974309+0000","last_change":"2024-03-18T14:43:41.982880+0000","last_active":"2024-03-18T14:43:42.974309+0000","last_peered":"2024-03-18T14:43:42.974309+0000","last_clean":"2024-03-18T14:43:42.974309+0000","last_became_active":"2024-03-18T14:43:41.982614+0000","last_became_peered":"2024-03-18T14:43:41.982614+0000","last_unstale":"2024-03-18T14:43:42.974309+0000","last_undegraded":"2024-03-18T14:43:42.974309+0000","last_fullsized":"2024-03-18T14:43:42.974309+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:15.827363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973837+0000","last_change":"2024-03-18T14:43:40.984498+0000","last_active":"2024-03-18T14:43:42.973837+0000","last_peered":"2024-03-18T14:43:42.973837+0000","last_clean":"2024-03-18T14:43:42.973837+0000","last_became_active":"2024-03-18T14:43:40.984005+0000","last_became_peered":"2024-03-18T14:43:40.984005+0000","last_unstale":"2024-03-18T14:43:42.973837+0000","last_undegraded":"2024-03-18T14:43:42.973837+0000","last_fullsized":"2024-03-18T14:43:42.973837+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:46.300184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970137+0000","last_change":"2024-03-18T14:43:41.987804+0000","last_active":"2024-03-18T14:43:42.970137+0000","last_peered":"2024-03-18T14:43:42.970137+0000","last_clean":"2024-03-18T14:43:42.970137+0000","last_became_active":"2024-03-18T14:43:41.987502+0000","last_became_peered":"2024-03-18T14:43:41.987502+0000","last_unstale":"2024-03-18T14:43:42.970137+0000","last_undegraded":"2024-03-18T14:43:42.970137+0000","last_fullsized":"2024-03-18T14:43:42.970137+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.004568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974399+0000","last_change":"2024-03-18T14:43:40.990752+0000","last_active":"2024-03-18T14:43:42.974399+0000","last_peered":"2024-03-18T14:43:42.974399+0000","last_clean":"2024-03-18T14:43:42.974399+0000","last_became_active":"2024-03-18T14:43:40.990438+0000","last_became_peered":"2024-03-18T14:43:40.990438+0000","last_unstale":"2024-03-18T14:43:42.974399+0000","last_undegraded":"2024-03-18T14:43:42.974399+0000","last_fullsized":"2024-03-18T14:43:42.974399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:20.609968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974336+0000","last_change":"2024-03-18T14:43:41.993639+0000","last_active":"2024-03-18T14:43:42.974336+0000","last_peered":"2024-03-18T14:43:42.974336+0000","last_clean":"2024-03-18T14:43:42.974336+0000","last_became_active":"2024-03-18T14:43:41.993353+0000","last_became_peered":"2024-03-18T14:43:41.993353+0000","last_unstale":"2024-03-18T14:43:42.974336+0000","last_undegraded":"2024-03-18T14:43:42.974336+0000","last_fullsized":"2024-03-18T14:43:42.974336+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:56.243015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975063+0000","last_change":"2024-03-18T14:43:40.983867+0000","last_active":"2024-03-18T14:43:42.975063+0000","last_peered":"2024-03-18T14:43:42.975063+0000","last_clean":"2024-03-18T14:43:42.975063+0000","last_became_active":"2024-03-18T14:43:40.983577+0000","last_became_peered":"2024-03-18T14:43:40.983577+0000","last_unstale":"2024-03-18T14:43:42.975063+0000","last_undegraded":"2024-03-18T14:43:42.975063+0000","last_fullsized":"2024-03-18T14:43:42.975063+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:45.865942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973430+0000","last_change":"2024-03-18T14:43:41.999442+0000","last_active":"2024-03-18T14:43:42.973430+0000","last_peered":"2024-03-18T14:43:42.973430+0000","last_clean":"2024-03-18T14:43:42.973430+0000","last_became_active":"2024-03-18T14:43:41.999197+0000","last_became_peered":"2024-03-18T14:43:41.999197+0000","last_unstale":"2024-03-18T14:43:42.973430+0000","last_undegraded":"2024-03-18T14:43:42.973430+0000","last_fullsized":"2024-03-18T14:43:42.973430+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:38:56.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973637+0000","last_change":"2024-03-18T14:43:40.987950+0000","last_active":"2024-03-18T14:43:42.973637+0000","last_peered":"2024-03-18T14:43:42.973637+0000","last_clean":"2024-03-18T14:43:42.973637+0000","last_became_active":"2024-03-18T14:43:40.987677+0000","last_became_peered":"2024-03-18T14:43:40.987677+0000","last_unstale":"2024-03-18T14:43:42.973637+0000","last_undegraded":"2024-03-18T14:43:42.973637+0000","last_fullsized":"2024-03-18T14:43:42.973637+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:33.490428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974172+0000","last_change":"2024-03-18T14:43:41.986125+0000","last_active":"2024-03-18T14:43:42.974172+0000","last_peered":"2024-03-18T14:43:42.974172+0000","last_clean":"2024-03-18T14:43:42.974172+0000","last_became_active":"2024-03-18T14:43:41.985878+0000","last_became_peered":"2024-03-18T14:43:41.985878+0000","last_unstale":"2024-03-18T14:43:42.974172+0000","last_undegraded":"2024-03-18T14:43:42.974172+0000","last_fullsized":"2024-03-18T14:43:42.974172+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:16.536785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971072+0000","last_change":"2024-03-18T14:43:40.985906+0000","last_active":"2024-03-18T14:43:42.971072+0000","last_peered":"2024-03-18T14:43:42.971072+0000","last_clean":"2024-03-18T14:43:42.971072+0000","last_became_active":"2024-03-18T14:43:40.985591+0000","last_became_peered":"2024-03-18T14:43:40.985591+0000","last_unstale":"2024-03-18T14:43:42.971072+0000","last_undegraded":"2024-03-18T14:43:42.971072+0000","last_fullsized":"2024-03-18T14:43:42.971072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:27.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974742+0000","last_change":"2024-03-18T14:43:41.984089+0000","last_active":"2024-03-18T14:43:42.974742+0000","last_peered":"2024-03-18T14:43:42.974742+0000","last_clean":"2024-03-18T14:43:42.974742+0000","last_became_active":"2024-03-18T14:43:41.983697+0000","last_became_peered":"2024-03-18T14:43:41.983697+0000","last_unstale":"2024-03-18T14:43:42.974742+0000","last_undegraded":"2024-03-18T14:43:42.974742+0000","last_fullsized":"2024-03-18T14:43:42.974742+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:18:24.577769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973801+0000","last_change":"2024-03-18T14:43:40.981949+0000","last_active":"2024-03-18T14:43:42.973801+0000","last_peered":"2024-03-18T14:43:42.973801+0000","last_clean":"2024-03-18T14:43:42.973801+0000","last_became_active":"2024-03-18T14:43:40.981625+0000","last_became_peered":"2024-03-18T14:43:40.981625+0000","last_unstale":"2024-03-18T14:43:42.973801+0000","last_undegraded":"2024-03-18T14:43:42.973801+0000","last_fullsized":"2024-03-18T14:43:42.973801+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:39:45.521393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974170+0000","last_change":"2024-03-18T14:43:41.989933+0000","last_active":"2024-03-18T14:43:42.974170+0000","last_peered":"2024-03-18T14:43:42.974170+0000","last_clean":"2024-03-18T14:43:42.974170+0000","last_became_active":"2024-03-18T14:43:41.988547+0000","last_became_peered":"2024-03-18T14:43:41.988547+0000","last_unstale":"2024-03-18T14:43:42.974170+0000","last_undegraded":"2024-03-18T14:43:42.974170+0000","last_fullsized":"2024-03-18T14:43:42.974170+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:39.379173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973696+0000","last_change":"2024-03-18T14:43:40.991385+0000","last_active":"2024-03-18T14:43:42.973696+0000","last_peered":"2024-03-18T14:43:42.973696+0000","last_clean":"2024-03-18T14:43:42.973696+0000","last_became_active":"2024-03-18T14:43:40.991010+0000","last_became_peered":"2024-03-18T14:43:40.991010+0000","last_unstale":"2024-03-18T14:43:42.973696+0000","last_undegraded":"2024-03-18T14:43:42.973696+0000","last_fullsized":"2024-03-18T14:43:42.973696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:28:29.280355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974175+0000","last_change":"2024-03-18T14:43:41.993861+0000","last_active":"2024-03-18T14:43:42.974175+0000","last_peered":"2024-03-18T14:43:42.974175+0000","last_clean":"2024-03-18T14:43:42.974175+0000","last_became_active":"2024-03-18T14:43:41.993590+0000","last_became_peered":"2024-03-18T14:43:41.993590+0000","last_unstale":"2024-03-18T14:43:42.974175+0000","last_undegraded":"2024-03-18T14:43:42.974175+0000","last_fullsized":"2024-03-18T14:43:42.974175+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:38:56.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974121+0000","last_change":"2024-03-18T14:43:40.982412+0000","last_active":"2024-03-18T14:43:42.974121+0000","last_peered":"2024-03-18T14:43:42.974121+0000","last_clean":"2024-03-18T14:43:42.974121+0000","last_became_active":"2024-03-18T14:43:40.982153+0000","last_became_peered":"2024-03-18T14:43:40.982153+0000","last_unstale":"2024-03-18T14:43:42.974121+0000","last_undegraded":"2024-03-18T14:43:42.974121+0000","last_fullsized":"2024-03-18T14:43:42.974121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:44.651178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973809+0000","last_change":"2024-03-18T14:43:41.998734+0000","last_active":"2024-03-18T14:43:42.973809+0000","last_peered":"2024-03-18T14:43:42.973809+0000","last_clean":"2024-03-18T14:43:42.973809+0000","last_became_active":"2024-03-18T14:43:41.998445+0000","last_became_peered":"2024-03-18T14:43:41.998445+0000","last_unstale":"2024-03-18T14:43:42.973809+0000","last_undegraded":"2024-03-18T14:43:42.973809+0000","last_fullsized":"2024-03-18T14:43:42.973809+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:15.827363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971724+0000","last_change":"2024-03-18T14:43:40.986458+0000","last_active":"2024-03-18T14:43:42.971724+0000","last_peered":"2024-03-18T14:43:42.971724+0000","last_clean":"2024-03-18T14:43:42.971724+0000","last_became_active":"2024-03-18T14:43:40.986161+0000","last_became_peered":"2024-03-18T14:43:40.986161+0000","last_unstale":"2024-03-18T14:43:42.971724+0000","last_undegraded":"2024-03-18T14:43:42.971724+0000","last_fullsized":"2024-03-18T14:43:42.971724+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:20:46.331310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973754+0000","last_change":"2024-03-18T14:43:41.988040+0000","last_active":"2024-03-18T14:43:42.973754+0000","last_peered":"2024-03-18T14:43:42.973754+0000","last_clean":"2024-03-18T14:43:42.973754+0000","last_became_active":"2024-03-18T14:43:41.987758+0000","last_became_peered":"2024-03-18T14:43:41.987758+0000","last_unstale":"2024-03-18T14:43:42.973754+0000","last_undegraded":"2024-03-18T14:43:42.973754+0000","last_fullsized":"2024-03-18T14:43:42.973754+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:13.115507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973614+0000","last_change":"2024-03-18T14:43:40.986938+0000","last_active":"2024-03-18T14:43:42.973614+0000","last_peered":"2024-03-18T14:43:42.973614+0000","last_clean":"2024-03-18T14:43:42.973614+0000","last_became_active":"2024-03-18T14:43:40.986666+0000","last_became_peered":"2024-03-18T14:43:40.986666+0000","last_unstale":"2024-03-18T14:43:42.973614+0000","last_undegraded":"2024-03-18T14:43:42.973614+0000","last_fullsized":"2024-03-18T14:43:42.973614+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:33:22.924944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974446+0000","last_change":"2024-03-18T14:43:41.993435+0000","last_active":"2024-03-18T14:43:42.974446+0000","last_peered":"2024-03-18T14:43:42.974446+0000","last_clean":"2024-03-18T14:43:42.974446+0000","last_became_active":"2024-03-18T14:43:41.993185+0000","last_became_peered":"2024-03-18T14:43:41.993185+0000","last_unstale":"2024-03-18T14:43:42.974446+0000","last_undegraded":"2024-03-18T14:43:42.974446+0000","last_fullsized":"2024-03-18T14:43:42.974446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:08.711617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971574+0000","last_change":"2024-03-18T14:43:40.989351+0000","last_active":"2024-03-18T14:43:42.971574+0000","last_peered":"2024-03-18T14:43:42.971574+0000","last_clean":"2024-03-18T14:43:42.971574+0000","last_became_active":"2024-03-18T14:43:40.989054+0000","last_became_peered":"2024-03-18T14:43:40.989054+0000","last_unstale":"2024-03-18T14:43:42.971574+0000","last_undegraded":"2024-03-18T14:43:42.971574+0000","last_fullsized":"2024-03-18T14:43:42.971574+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:59.144538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973482+0000","last_change":"2024-03-18T14:43:41.994745+0000","last_active":"2024-03-18T14:43:42.973482+0000","last_peered":"2024-03-18T14:43:42.973482+0000","last_clean":"2024-03-18T14:43:42.973482+0000","last_became_active":"2024-03-18T14:43:41.993838+0000","last_became_peered":"2024-03-18T14:43:41.993838+0000","last_unstale":"2024-03-18T14:43:42.973482+0000","last_undegraded":"2024-03-18T14:43:42.973482+0000","last_fullsized":"2024-03-18T14:43:42.973482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:16.536785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973886+0000","last_change":"2024-03-18T14:43:40.982844+0000","last_active":"2024-03-18T14:43:42.973886+0000","last_peered":"2024-03-18T14:43:42.973886+0000","last_clean":"2024-03-18T14:43:42.973886+0000","last_became_active":"2024-03-18T14:43:40.982535+0000","last_became_peered":"2024-03-18T14:43:40.982535+0000","last_unstale":"2024-03-18T14:43:42.973886+0000","last_undegraded":"2024-03-18T14:43:42.973886+0000","last_fullsized":"2024-03-18T14:43:42.973886+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:49:51.094883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971320+0000","last_change":"2024-03-18T14:43:41.994275+0000","last_active":"2024-03-18T14:43:42.971320+0000","last_peered":"2024-03-18T14:43:42.971320+0000","last_clean":"2024-03-18T14:43:42.971320+0000","last_became_active":"2024-03-18T14:43:41.993874+0000","last_became_peered":"2024-03-18T14:43:41.993874+0000","last_unstale":"2024-03-18T14:43:42.971320+0000","last_undegraded":"2024-03-18T14:43:42.971320+0000","last_fullsized":"2024-03-18T14:43:42.971320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:56.243015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970214+0000","last_change":"2024-03-18T14:43:40.982113+0000","last_active":"2024-03-18T14:43:42.970214+0000","last_peered":"2024-03-18T14:43:42.970214+0000","last_clean":"2024-03-18T14:43:42.970214+0000","last_became_active":"2024-03-18T14:43:40.981369+0000","last_became_peered":"2024-03-18T14:43:40.981369+0000","last_unstale":"2024-03-18T14:43:42.970214+0000","last_undegraded":"2024-03-18T14:43:42.970214+0000","last_fullsized":"2024-03-18T14:43:42.970214+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:53.144792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970926+0000","last_change":"2024-03-18T14:43:41.993255+0000","last_active":"2024-03-18T14:43:42.970926+0000","last_peered":"2024-03-18T14:43:42.970926+0000","last_clean":"2024-03-18T14:43:42.970926+0000","last_became_active":"2024-03-18T14:43:41.992976+0000","last_became_peered":"2024-03-18T14:43:41.992976+0000","last_unstale":"2024-03-18T14:43:42.970926+0000","last_undegraded":"2024-03-18T14:43:42.970926+0000","last_fullsized":"2024-03-18T14:43:42.970926+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:16.536785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973541+0000","last_change":"2024-03-18T14:43:40.989232+0000","last_active":"2024-03-18T14:43:42.973541+0000","last_peered":"2024-03-18T14:43:42.973541+0000","last_clean":"2024-03-18T14:43:42.973541+0000","last_became_active":"2024-03-18T14:43:40.988982+0000","last_became_peered":"2024-03-18T14:43:40.988982+0000","last_unstale":"2024-03-18T14:43:42.973541+0000","last_undegraded":"2024-03-18T14:43:42.973541+0000","last_fullsized":"2024-03-18T14:43:42.973541+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:20.541410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973864+0000","last_change":"2024-03-18T14:43:41.987124+0000","last_active":"2024-03-18T14:43:42.973864+0000","last_peered":"2024-03-18T14:43:42.973864+0000","last_clean":"2024-03-18T14:43:42.973864+0000","last_became_active":"2024-03-18T14:43:41.986518+0000","last_became_peered":"2024-03-18T14:43:41.986518+0000","last_unstale":"2024-03-18T14:43:42.973864+0000","last_undegraded":"2024-03-18T14:43:42.973864+0000","last_fullsized":"2024-03-18T14:43:42.973864+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:56.329557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970407+0000","last_change":"2024-03-18T14:43:40.988468+0000","last_active":"2024-03-18T14:43:42.970407+0000","last_peered":"2024-03-18T14:43:42.970407+0000","last_clean":"2024-03-18T14:43:42.970407+0000","last_became_active":"2024-03-18T14:43:40.988130+0000","last_became_peered":"2024-03-18T14:43:40.988130+0000","last_unstale":"2024-03-18T14:43:42.970407+0000","last_undegraded":"2024-03-18T14:43:42.970407+0000","last_fullsized":"2024-03-18T14:43:42.970407+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:27.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973978+0000","last_change":"2024-03-18T14:43:41.990918+0000","last_active":"2024-03-18T14:43:42.973978+0000","last_peered":"2024-03-18T14:43:42.973978+0000","last_clean":"2024-03-18T14:43:42.973978+0000","last_became_active":"2024-03-18T14:43:41.990598+0000","last_became_peered":"2024-03-18T14:43:41.990598+0000","last_unstale":"2024-03-18T14:43:42.973978+0000","last_undegraded":"2024-03-18T14:43:42.973978+0000","last_fullsized":"2024-03-18T14:43:42.973978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:56.243015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.986543+0000","last_change":"2024-03-18T14:43:40.984516+0000","last_active":"2024-03-18T14:43:42.986543+0000","last_peered":"2024-03-18T14:43:42.986543+0000","last_clean":"2024-03-18T14:43:42.986543+0000","last_became_active":"2024-03-18T14:43:40.984037+0000","last_became_peered":"2024-03-18T14:43:40.984037+0000","last_unstale":"2024-03-18T14:43:42.986543+0000","last_undegraded":"2024-03-18T14:43:42.986543+0000","last_fullsized":"2024-03-18T14:43:42.986543+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:20.541410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970269+0000","last_change":"2024-03-18T14:43:41.984502+0000","last_active":"2024-03-18T14:43:42.970269+0000","last_peered":"2024-03-18T14:43:42.970269+0000","last_clean":"2024-03-18T14:43:42.970269+0000","last_became_active":"2024-03-18T14:43:41.984186+0000","last_became_peered":"2024-03-18T14:43:41.984186+0000","last_unstale":"2024-03-18T14:43:42.970269+0000","last_undegraded":"2024-03-18T14:43:42.970269+0000","last_fullsized":"2024-03-18T14:43:42.970269+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:16.791852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971307+0000","last_change":"2024-03-18T14:43:40.979968+0000","last_active":"2024-03-18T14:43:42.971307+0000","last_peered":"2024-03-18T14:43:42.971307+0000","last_clean":"2024-03-18T14:43:42.971307+0000","last_became_active":"2024-03-18T14:43:40.979691+0000","last_became_peered":"2024-03-18T14:43:40.979691+0000","last_unstale":"2024-03-18T14:43:42.971307+0000","last_undegraded":"2024-03-18T14:43:42.971307+0000","last_fullsized":"2024-03-18T14:43:42.971307+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:36:26.213692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971641+0000","last_change":"2024-03-18T14:43:41.989816+0000","last_active":"2024-03-18T14:43:42.971641+0000","last_peered":"2024-03-18T14:43:42.971641+0000","last_clean":"2024-03-18T14:43:42.971641+0000","last_became_active":"2024-03-18T14:43:41.989471+0000","last_became_peered":"2024-03-18T14:43:41.989471+0000","last_unstale":"2024-03-18T14:43:42.971641+0000","last_undegraded":"2024-03-18T14:43:42.971641+0000","last_fullsized":"2024-03-18T14:43:42.971641+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:22.376659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974585+0000","last_change":"2024-03-18T14:43:40.982991+0000","last_active":"2024-03-18T14:43:42.974585+0000","last_peered":"2024-03-18T14:43:42.974585+0000","last_clean":"2024-03-18T14:43:42.974585+0000","last_became_active":"2024-03-18T14:43:40.982709+0000","last_became_peered":"2024-03-18T14:43:40.982709+0000","last_unstale":"2024-03-18T14:43:42.974585+0000","last_undegraded":"2024-03-18T14:43:42.974585+0000","last_fullsized":"2024-03-18T14:43:42.974585+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:57.222848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973617+0000","last_change":"2024-03-18T14:43:41.976076+0000","last_active":"2024-03-18T14:43:42.973617+0000","last_peered":"2024-03-18T14:43:42.973617+0000","last_clean":"2024-03-18T14:43:42.973617+0000","last_became_active":"2024-03-18T14:43:41.975418+0000","last_became_peered":"2024-03-18T14:43:41.975418+0000","last_unstale":"2024-03-18T14:43:42.973617+0000","last_undegraded":"2024-03-18T14:43:42.973617+0000","last_fullsized":"2024-03-18T14:43:42.973617+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:37:54.128119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.969987+0000","last_change":"2024-03-18T14:43:40.979481+0000","last_active":"2024-03-18T14:43:42.969987+0000","last_peered":"2024-03-18T14:43:42.969987+0000","last_clean":"2024-03-18T14:43:42.969987+0000","last_became_active":"2024-03-18T14:43:40.979205+0000","last_became_peered":"2024-03-18T14:43:40.979205+0000","last_unstale":"2024-03-18T14:43:42.969987+0000","last_undegraded":"2024-03-18T14:43:42.969987+0000","last_fullsized":"2024-03-18T14:43:42.969987+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:46.392263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974059+0000","last_change":"2024-03-18T14:43:41.983354+0000","last_active":"2024-03-18T14:43:42.974059+0000","last_peered":"2024-03-18T14:43:42.974059+0000","last_clean":"2024-03-18T14:43:42.974059+0000","last_became_active":"2024-03-18T14:43:41.983072+0000","last_became_peered":"2024-03-18T14:43:41.983072+0000","last_unstale":"2024-03-18T14:43:42.974059+0000","last_undegraded":"2024-03-18T14:43:42.974059+0000","last_fullsized":"2024-03-18T14:43:42.974059+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:26:47.695549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975063+0000","last_change":"2024-03-18T14:43:40.984587+0000","last_active":"2024-03-18T14:43:42.975063+0000","last_peered":"2024-03-18T14:43:42.975063+0000","last_clean":"2024-03-18T14:43:42.975063+0000","last_became_active":"2024-03-18T14:43:40.984301+0000","last_became_peered":"2024-03-18T14:43:40.984301+0000","last_unstale":"2024-03-18T14:43:42.975063+0000","last_undegraded":"2024-03-18T14:43:42.975063+0000","last_fullsized":"2024-03-18T14:43:42.975063+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:59.274683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.969891+0000","last_change":"2024-03-18T14:43:41.986909+0000","last_active":"2024-03-18T14:43:42.969891+0000","last_peered":"2024-03-18T14:43:42.969891+0000","last_clean":"2024-03-18T14:43:42.969891+0000","last_became_active":"2024-03-18T14:43:41.986636+0000","last_became_peered":"2024-03-18T14:43:41.986636+0000","last_unstale":"2024-03-18T14:43:42.969891+0000","last_undegraded":"2024-03-18T14:43:42.969891+0000","last_fullsized":"2024-03-18T14:43:42.969891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:33.841740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.969997+0000","last_change":"2024-03-18T14:43:40.978243+0000","last_active":"2024-03-18T14:43:42.969997+0000","last_peered":"2024-03-18T14:43:42.969997+0000","last_clean":"2024-03-18T14:43:42.969997+0000","last_became_active":"2024-03-18T14:43:40.977980+0000","last_became_peered":"2024-03-18T14:43:40.977980+0000","last_unstale":"2024-03-18T14:43:42.969997+0000","last_undegraded":"2024-03-18T14:43:42.969997+0000","last_fullsized":"2024-03-18T14:43:42.969997+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:02.875572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973705+0000","last_change":"2024-03-18T14:43:41.988993+0000","last_active":"2024-03-18T14:43:42.973705+0000","last_peered":"2024-03-18T14:43:42.973705+0000","last_clean":"2024-03-18T14:43:42.973705+0000","last_became_active":"2024-03-18T14:43:41.988308+0000","last_became_peered":"2024-03-18T14:43:41.988308+0000","last_unstale":"2024-03-18T14:43:42.973705+0000","last_undegraded":"2024-03-18T14:43:42.973705+0000","last_fullsized":"2024-03-18T14:43:42.973705+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:24.113873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973396+0000","last_change":"2024-03-18T14:43:40.979654+0000","last_active":"2024-03-18T14:43:42.973396+0000","last_peered":"2024-03-18T14:43:42.973396+0000","last_clean":"2024-03-18T14:43:42.973396+0000","last_became_active":"2024-03-18T14:43:40.979137+0000","last_became_peered":"2024-03-18T14:43:40.979137+0000","last_unstale":"2024-03-18T14:43:42.973396+0000","last_undegraded":"2024-03-18T14:43:42.973396+0000","last_fullsized":"2024-03-18T14:43:42.973396+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:46:09.227198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974070+0000","last_change":"2024-03-18T14:43:41.986366+0000","last_active":"2024-03-18T14:43:42.974070+0000","last_peered":"2024-03-18T14:43:42.974070+0000","last_clean":"2024-03-18T14:43:42.974070+0000","last_became_active":"2024-03-18T14:43:41.986082+0000","last_became_peered":"2024-03-18T14:43:41.986082+0000","last_unstale":"2024-03-18T14:43:42.974070+0000","last_undegraded":"2024-03-18T14:43:42.974070+0000","last_fullsized":"2024-03-18T14:43:42.974070+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:14:46.090742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971131+0000","last_change":"2024-03-18T14:43:40.978079+0000","last_active":"2024-03-18T14:43:42.971131+0000","last_peered":"2024-03-18T14:43:42.971131+0000","last_clean":"2024-03-18T14:43:42.971131+0000","last_became_active":"2024-03-18T14:43:40.977784+0000","last_became_peered":"2024-03-18T14:43:40.977784+0000","last_unstale":"2024-03-18T14:43:42.971131+0000","last_undegraded":"2024-03-18T14:43:42.971131+0000","last_fullsized":"2024-03-18T14:43:42.971131+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:53.144792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":233,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973747+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:43:42.973747+0000","last_peered":"2024-03-18T14:43:42.973747+0000","last_clean":"2024-03-18T14:43:42.973747+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:43:42.973747+0000","last_undegraded":"2024-03-18T14:43:42.973747+0000","last_fullsized":"2024-03-18T14:43:42.973747+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":297457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":894677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":305761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":667648,"data_stored":297513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":663552,"data_stored":297547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1269760,"data_stored":890019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:43:45.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:43:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.127+0000 7fc7970ff640 1 Processor -- start 2024-03-18T14:43:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.127+0000 7fc7970ff640 1 -- start start 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.127+0000 7fc7970ff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc79015e020 0x7fc790162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc7970ff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7900b9660 con 0x7fc7900ced00 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc7970ff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7900ce390 con 0x7fc7900d3210 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc7970ff640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc7900d2270 con 0x7fc79015e020 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc7900ced00 0x7fc7900d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58090/0 (socket says 172.21.15.6:58090) 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78ffff640 1 -- 172.21.15.6:0/2002945982 learned_addr learned my addr 172.21.15.6:0/2002945982 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc794e9b640 1 --2- 172.21.15.6:0/2002945982 >> v2:172.21.15.92:3300/0 conn(0x7fc79015e020 0x7fc790162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516477747 0 0) 0x7fc7900b9660 con 0x7fc7900ced00 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc780003520 con 0x7fc7900ced00 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1053465958 0 0) 0x7fc780003520 con 0x7fc7900ced00 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 >> v2:172.21.15.92:3300/0 conn(0x7fc79015e020 msgr2=0x7fc790162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 --2- 172.21.15.6:0/2002945982 >> v2:172.21.15.92:3300/0 conn(0x7fc79015e020 0x7fc790162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 >> v1:172.21.15.92:6789/0 conn(0x7fc7900d3210 legacy=0x7fc7900d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7900b9900 con 0x7fc7900ced00 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc794e9b640 1 --2- 172.21.15.6:0/2002945982 >> v2:172.21.15.92:3300/0 conn(0x7fc79015e020 0x7fc790162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fc77c001690 con 0x7fc7900ced00 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 >> v1:172.21.15.6:6789/0 conn(0x7fc7900ced00 legacy=0x7fc7900d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7800039b0 0x7fc780003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc780004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7800051c0 0x7fc78000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc7900d2270 con 0x7fc780004800 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc78000c160 con 0x7fc7800039b0 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc78000c360 con 0x7fc7800051c0 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.128+0000 7fc78ffff640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc780004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc794e9b640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7800039b0 0x7fc780003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc79569c640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7800051c0 0x7fc78000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc78ffff640 1 -- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7800051c0 msgr2=0x7fc78000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc78ffff640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7800051c0 0x7fc78000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc78ffff640 1 -- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7800039b0 msgr2=0x7fc780003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc78ffff640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7800039b0 0x7fc780003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc78ffff640 1 -- 172.21.15.6:0/2002945982 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc780003520 con 0x7fc780004800 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc79569c640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7800051c0 0x7fc78000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.129+0000 7fc79569c640 1 -- 172.21.15.6:0/2002945982 reap_dead start 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc78ffff640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc780004c80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c210 tx=0x7fc780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52871e86ffd9d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c007d30 con 0x7fc780004800 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc77c030070 con 0x7fc780004800 2024-03-18T14:43:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc78f7fe640 1 -- 172.21.15.6:0/2002945982 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c005d30 con 0x7fc780004800 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc7970ff640 1 -- 172.21.15.6:0/2002945982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 msgr2=0x7fc780004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc7970ff640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc780004c80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c210 tx=0x7fc780005830 comp rx=0 tx=0).stop 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc7970ff640 1 -- 172.21.15.6:0/2002945982 shutdown_connections 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc7970ff640 1 --2- 172.21.15.6:0/2002945982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc780004c80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc7970ff640 1 -- 172.21.15.6:0/2002945982 >> 172.21.15.6:0/2002945982 conn(0x7fc790072ce0 msgr2=0x7fc7900730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc7970ff640 1 -- 172.21.15.6:0/2002945982 shutdown_connections 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.130+0000 7fc7970ff640 1 -- 172.21.15.6:0/2002945982 wait complete. 2024-03-18T14:43:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.131+0000 7fc7970ff640 1 Processor -- start 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.131+0000 7fc7970ff640 1 -- start start 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc7970ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc790154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc7970ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc79015e570 0x7fc7900d3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc7970ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc790153400 0x7fc7900c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc7970ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc7900cf2c0 con 0x7fc780004800 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc7970ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc7900ce890 con 0x7fc79015e570 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc7970ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc7900cf440 con 0x7fc790153400 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc794e9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc790154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc794e9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc790154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36036/0 (socket says 172.21.15.6:36036) 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.132+0000 7fc794e9b640 1 -- 172.21.15.6:0/1721296754 learned_addr learned my addr 172.21.15.6:0/1721296754 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc79569c640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc790153400 0x7fc7900c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc78ffff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc79015e570 0x7fc7900d3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc794e9b640 1 -- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc790153400 msgr2=0x7fc7900c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc794e9b640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc790153400 0x7fc7900c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc794e9b640 1 -- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc79015e570 msgr2=0x7fc7900d3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc794e9b640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc79015e570 0x7fc7900d3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc794e9b640 1 -- 172.21.15.6:0/1721296754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc790153e30 con 0x7fc780004800 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc794e9b640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc790154e80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc784011cd0 tx=0x7fc78400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc78d7fa640 1 -- 172.21.15.6:0/1721296754 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78400fd20 con 0x7fc780004800 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc78d7fa640 1 -- 172.21.15.6:0/1721296754 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc784004020 con 0x7fc780004800 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc790154120 con 0x7fc780004800 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc78d7fa640 1 -- 172.21.15.6:0/1721296754 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784004410 con 0x7fc780004800 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.133+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc790154960 con 0x7fc780004800 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.134+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79008afe0 con 0x7fc780004800 2024-03-18T14:43:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.135+0000 7fc78d7fa640 1 -- 172.21.15.6:0/1721296754 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc784012040 con 0x7fc780004800 2024-03-18T14:43:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.135+0000 7fc78d7fa640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc77406bbd0 0x7fc77406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.135+0000 7fc78d7fa640 1 -- 172.21.15.6:0/1721296754 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fc78408b1f0 con 0x7fc780004800 2024-03-18T14:43:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.135+0000 7fc78ffff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc77406bbd0 0x7fc77406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.136+0000 7fc78ffff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc77406bbd0 0x7fc77406e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc77c034000 tx=0x7fc77c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.139+0000 7fc78d7fa640 1 -- 172.21.15.6:0/1721296754 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc790154120 con 0x7fc780004800 2024-03-18T14:43:46.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.392+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc79004f620 con 0x7fc77406bbd0 2024-03-18T14:43:46.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.399+0000 7fc78d7fa640 1 -- 172.21.15.6:0/1721296754 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251954 (secure 0 0 0) 0x7fc79004f620 con 0x7fc77406bbd0 2024-03-18T14:43:46.404 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:46.405 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:43:46.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc77406bbd0 msgr2=0x7fc77406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc77406bbd0 0x7fc77406e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc77c034000 tx=0x7fc77c009030 comp rx=0 tx=0).stop 2024-03-18T14:43:46.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 msgr2=0x7fc790154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc790154e80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc784011cd0 tx=0x7fc78400f610 comp rx=0 tx=0).stop 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 shutdown_connections 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc77406bbd0 0x7fc77406e090 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc790153400 0x7fc7900c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc79015e570 0x7fc7900d3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 --2- 172.21.15.6:0/1721296754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc780004800 0x7fc790154e80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 >> 172.21.15.6:0/1721296754 conn(0x7fc790072ce0 msgr2=0x7fc790161e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 shutdown_connections 2024-03-18T14:43:46.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.402+0000 7fc7970ff640 1 -- 172.21.15.6:0/1721296754 wait complete. 2024-03-18T14:43:46.416 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-03-18T14:43:45.408569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222604,"kb_used_data":6424,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752116,"statfs":{"total":773094113280,"available":772866166784,"internally_reserved":0,"allocated":6578176,"data_stored":3591980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.448003"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974008+0000","last_change":"2024-03-18T14:43:41.978141+0000","last_active":"2024-03-18T14:43:42.974008+0000","last_peered":"2024-03-18T14:43:42.974008+0000","last_clean":"2024-03-18T14:43:42.974008+0000","last_became_active":"2024-03-18T14:43:41.977839+0000","last_became_peered":"2024-03-18T14:43:41.977839+0000","last_unstale":"2024-03-18T14:43:42.974008+0000","last_undegraded":"2024-03-18T14:43:42.974008+0000","last_fullsized":"2024-03-18T14:43:42.974008+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:49:20.830857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973743+0000","last_change":"2024-03-18T14:43:41.976097+0000","last_active":"2024-03-18T14:43:42.973743+0000","last_peered":"2024-03-18T14:43:42.973743+0000","last_clean":"2024-03-18T14:43:42.973743+0000","last_became_active":"2024-03-18T14:43:41.975825+0000","last_became_peered":"2024-03-18T14:43:41.975825+0000","last_unstale":"2024-03-18T14:43:42.973743+0000","last_undegraded":"2024-03-18T14:43:42.973743+0000","last_fullsized":"2024-03-18T14:43:42.973743+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:45:33.774134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974306+0000","last_change":"2024-03-18T14:43:41.988970+0000","last_active":"2024-03-18T14:43:42.974306+0000","last_peered":"2024-03-18T14:43:42.974306+0000","last_clean":"2024-03-18T14:43:42.974306+0000","last_became_active":"2024-03-18T14:43:41.988696+0000","last_became_peered":"2024-03-18T14:43:41.988696+0000","last_unstale":"2024-03-18T14:43:42.974306+0000","last_undegraded":"2024-03-18T14:43:42.974306+0000","last_fullsized":"2024-03-18T14:43:42.974306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:56.329557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974477+0000","last_change":"2024-03-18T14:43:41.989221+0000","last_active":"2024-03-18T14:43:42.974477+0000","last_peered":"2024-03-18T14:43:42.974477+0000","last_clean":"2024-03-18T14:43:42.974477+0000","last_became_active":"2024-03-18T14:43:41.987976+0000","last_became_peered":"2024-03-18T14:43:41.987976+0000","last_unstale":"2024-03-18T14:43:42.974477+0000","last_undegraded":"2024-03-18T14:43:42.974477+0000","last_fullsized":"2024-03-18T14:43:42.974477+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:08.711617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973558+0000","last_change":"2024-03-18T14:43:41.987473+0000","last_active":"2024-03-18T14:43:42.973558+0000","last_peered":"2024-03-18T14:43:42.973558+0000","last_clean":"2024-03-18T14:43:42.973558+0000","last_became_active":"2024-03-18T14:43:41.987208+0000","last_became_peered":"2024-03-18T14:43:41.987208+0000","last_unstale":"2024-03-18T14:43:42.973558+0000","last_undegraded":"2024-03-18T14:43:42.973558+0000","last_fullsized":"2024-03-18T14:43:42.973558+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:47.711875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971160+0000","last_change":"2024-03-18T14:43:41.994240+0000","last_active":"2024-03-18T14:43:42.971160+0000","last_peered":"2024-03-18T14:43:42.971160+0000","last_clean":"2024-03-18T14:43:42.971160+0000","last_became_active":"2024-03-18T14:43:41.993537+0000","last_became_peered":"2024-03-18T14:43:41.993537+0000","last_unstale":"2024-03-18T14:43:42.971160+0000","last_undegraded":"2024-03-18T14:43:42.971160+0000","last_fullsized":"2024-03-18T14:43:42.971160+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:15.827363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974992+0000","last_change":"2024-03-18T14:43:41.988101+0000","last_active":"2024-03-18T14:43:42.974992+0000","last_peered":"2024-03-18T14:43:42.974992+0000","last_clean":"2024-03-18T14:43:42.974992+0000","last_became_active":"2024-03-18T14:43:41.987819+0000","last_became_peered":"2024-03-18T14:43:41.987819+0000","last_unstale":"2024-03-18T14:43:42.974992+0000","last_undegraded":"2024-03-18T14:43:42.974992+0000","last_fullsized":"2024-03-18T14:43:42.974992+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:21.618875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973298+0000","last_change":"2024-03-18T14:43:41.981904+0000","last_active":"2024-03-18T14:43:42.973298+0000","last_peered":"2024-03-18T14:43:42.973298+0000","last_clean":"2024-03-18T14:43:42.973298+0000","last_became_active":"2024-03-18T14:43:41.980691+0000","last_became_peered":"2024-03-18T14:43:41.980691+0000","last_unstale":"2024-03-18T14:43:42.973298+0000","last_undegraded":"2024-03-18T14:43:42.973298+0000","last_fullsized":"2024-03-18T14:43:42.973298+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:17:28.166306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975085+0000","last_change":"2024-03-18T14:43:41.989662+0000","last_active":"2024-03-18T14:43:42.975085+0000","last_peered":"2024-03-18T14:43:42.975085+0000","last_clean":"2024-03-18T14:43:42.975085+0000","last_became_active":"2024-03-18T14:43:41.989148+0000","last_became_peered":"2024-03-18T14:43:41.989148+0000","last_unstale":"2024-03-18T14:43:42.975085+0000","last_undegraded":"2024-03-18T14:43:42.975085+0000","last_fullsized":"2024-03-18T14:43:42.975085+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:20:47.340218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970250+0000","last_change":"2024-03-18T14:43:41.987080+0000","last_active":"2024-03-18T14:43:42.970250+0000","last_peered":"2024-03-18T14:43:42.970250+0000","last_clean":"2024-03-18T14:43:42.970250+0000","last_became_active":"2024-03-18T14:43:41.986789+0000","last_became_peered":"2024-03-18T14:43:41.986789+0000","last_unstale":"2024-03-18T14:43:42.970250+0000","last_undegraded":"2024-03-18T14:43:42.970250+0000","last_fullsized":"2024-03-18T14:43:42.970250+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:18:49.721963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970573+0000","last_change":"2024-03-18T14:43:41.976980+0000","last_active":"2024-03-18T14:43:42.970573+0000","last_peered":"2024-03-18T14:43:42.970573+0000","last_clean":"2024-03-18T14:43:42.970573+0000","last_became_active":"2024-03-18T14:43:41.976611+0000","last_became_peered":"2024-03-18T14:43:41.976611+0000","last_unstale":"2024-03-18T14:43:42.970573+0000","last_undegraded":"2024-03-18T14:43:42.970573+0000","last_fullsized":"2024-03-18T14:43:42.970573+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:28:30.289262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974494+0000","last_change":"2024-03-18T14:43:41.990208+0000","last_active":"2024-03-18T14:43:42.974494+0000","last_peered":"2024-03-18T14:43:42.974494+0000","last_clean":"2024-03-18T14:43:42.974494+0000","last_became_active":"2024-03-18T14:43:41.989913+0000","last_became_peered":"2024-03-18T14:43:41.989913+0000","last_unstale":"2024-03-18T14:43:42.974494+0000","last_undegraded":"2024-03-18T14:43:42.974494+0000","last_fullsized":"2024-03-18T14:43:42.974494+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:22.376659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971540+0000","last_change":"2024-03-18T14:43:41.991068+0000","last_active":"2024-03-18T14:43:42.971540+0000","last_peered":"2024-03-18T14:43:42.971540+0000","last_clean":"2024-03-18T14:43:42.971540+0000","last_became_active":"2024-03-18T14:43:41.990801+0000","last_became_peered":"2024-03-18T14:43:41.990801+0000","last_unstale":"2024-03-18T14:43:42.971540+0000","last_undegraded":"2024-03-18T14:43:42.971540+0000","last_fullsized":"2024-03-18T14:43:42.971540+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.004568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975089+0000","last_change":"2024-03-18T14:43:41.987107+0000","last_active":"2024-03-18T14:43:42.975089+0000","last_peered":"2024-03-18T14:43:42.975089+0000","last_clean":"2024-03-18T14:43:42.975089+0000","last_became_active":"2024-03-18T14:43:41.986829+0000","last_became_peered":"2024-03-18T14:43:41.986829+0000","last_unstale":"2024-03-18T14:43:42.975089+0000","last_undegraded":"2024-03-18T14:43:42.975089+0000","last_fullsized":"2024-03-18T14:43:42.975089+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:31.332178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974995+0000","last_change":"2024-03-18T14:43:41.992096+0000","last_active":"2024-03-18T14:43:42.974995+0000","last_peered":"2024-03-18T14:43:42.974995+0000","last_clean":"2024-03-18T14:43:42.974995+0000","last_became_active":"2024-03-18T14:43:41.991792+0000","last_became_peered":"2024-03-18T14:43:41.991792+0000","last_unstale":"2024-03-18T14:43:42.974995+0000","last_undegraded":"2024-03-18T14:43:42.974995+0000","last_fullsized":"2024-03-18T14:43:42.974995+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:37:00.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970872+0000","last_change":"2024-03-18T14:43:41.996226+0000","last_active":"2024-03-18T14:43:42.970872+0000","last_peered":"2024-03-18T14:43:42.970872+0000","last_clean":"2024-03-18T14:43:42.970872+0000","last_became_active":"2024-03-18T14:43:41.995906+0000","last_became_peered":"2024-03-18T14:43:41.995906+0000","last_unstale":"2024-03-18T14:43:42.970872+0000","last_undegraded":"2024-03-18T14:43:42.970872+0000","last_fullsized":"2024-03-18T14:43:42.970872+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:39.379173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974000+0000","last_change":"2024-03-18T14:43:41.991239+0000","last_active":"2024-03-18T14:43:42.974000+0000","last_peered":"2024-03-18T14:43:42.974000+0000","last_clean":"2024-03-18T14:43:42.974000+0000","last_became_active":"2024-03-18T14:43:41.990929+0000","last_became_peered":"2024-03-18T14:43:41.990929+0000","last_unstale":"2024-03-18T14:43:42.974000+0000","last_undegraded":"2024-03-18T14:43:42.974000+0000","last_fullsized":"2024-03-18T14:43:42.974000+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:44:34.079834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974248+0000","last_change":"2024-03-18T14:43:41.999148+0000","last_active":"2024-03-18T14:43:42.974248+0000","last_peered":"2024-03-18T14:43:42.974248+0000","last_clean":"2024-03-18T14:43:42.974248+0000","last_became_active":"2024-03-18T14:43:41.998903+0000","last_became_peered":"2024-03-18T14:43:41.998903+0000","last_unstale":"2024-03-18T14:43:42.974248+0000","last_undegraded":"2024-03-18T14:43:42.974248+0000","last_fullsized":"2024-03-18T14:43:42.974248+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:50.325632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973589+0000","last_change":"2024-03-18T14:43:41.978655+0000","last_active":"2024-03-18T14:43:42.973589+0000","last_peered":"2024-03-18T14:43:42.973589+0000","last_clean":"2024-03-18T14:43:42.973589+0000","last_became_active":"2024-03-18T14:43:41.977381+0000","last_became_peered":"2024-03-18T14:43:41.977381+0000","last_unstale":"2024-03-18T14:43:42.973589+0000","last_undegraded":"2024-03-18T14:43:42.973589+0000","last_fullsized":"2024-03-18T14:43:42.973589+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:06.678808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973610+0000","last_change":"2024-03-18T14:43:41.988740+0000","last_active":"2024-03-18T14:43:42.973610+0000","last_peered":"2024-03-18T14:43:42.973610+0000","last_clean":"2024-03-18T14:43:42.973610+0000","last_became_active":"2024-03-18T14:43:41.988378+0000","last_became_peered":"2024-03-18T14:43:41.988378+0000","last_unstale":"2024-03-18T14:43:42.973610+0000","last_undegraded":"2024-03-18T14:43:42.973610+0000","last_fullsized":"2024-03-18T14:43:42.973610+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:02:47.517991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973825+0000","last_change":"2024-03-18T14:43:41.989206+0000","last_active":"2024-03-18T14:43:42.973825+0000","last_peered":"2024-03-18T14:43:42.973825+0000","last_clean":"2024-03-18T14:43:42.973825+0000","last_became_active":"2024-03-18T14:43:41.988919+0000","last_became_peered":"2024-03-18T14:43:41.988919+0000","last_unstale":"2024-03-18T14:43:42.973825+0000","last_undegraded":"2024-03-18T14:43:42.973825+0000","last_fullsized":"2024-03-18T14:43:42.973825+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:49:52.103790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973982+0000","last_change":"2024-03-18T14:43:41.976324+0000","last_active":"2024-03-18T14:43:42.973982+0000","last_peered":"2024-03-18T14:43:42.973982+0000","last_clean":"2024-03-18T14:43:42.973982+0000","last_became_active":"2024-03-18T14:43:41.975861+0000","last_became_peered":"2024-03-18T14:43:41.975861+0000","last_unstale":"2024-03-18T14:43:42.973982+0000","last_undegraded":"2024-03-18T14:43:42.973982+0000","last_fullsized":"2024-03-18T14:43:42.973982+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:16.750519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973817+0000","last_change":"2024-03-18T14:43:41.987768+0000","last_active":"2024-03-18T14:43:42.973817+0000","last_peered":"2024-03-18T14:43:42.973817+0000","last_clean":"2024-03-18T14:43:42.973817+0000","last_became_active":"2024-03-18T14:43:41.987494+0000","last_became_peered":"2024-03-18T14:43:41.987494+0000","last_unstale":"2024-03-18T14:43:42.973817+0000","last_undegraded":"2024-03-18T14:43:42.973817+0000","last_fullsized":"2024-03-18T14:43:42.973817+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:34.499335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971158+0000","last_change":"2024-03-18T14:43:41.998908+0000","last_active":"2024-03-18T14:43:42.971158+0000","last_peered":"2024-03-18T14:43:42.971158+0000","last_clean":"2024-03-18T14:43:42.971158+0000","last_became_active":"2024-03-18T14:43:41.998510+0000","last_became_peered":"2024-03-18T14:43:41.998510+0000","last_unstale":"2024-03-18T14:43:42.971158+0000","last_undegraded":"2024-03-18T14:43:42.971158+0000","last_fullsized":"2024-03-18T14:43:42.971158+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:20.750238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974151+0000","last_change":"2024-03-18T14:43:41.977490+0000","last_active":"2024-03-18T14:43:42.974151+0000","last_peered":"2024-03-18T14:43:42.974151+0000","last_clean":"2024-03-18T14:43:42.974151+0000","last_became_active":"2024-03-18T14:43:41.977174+0000","last_became_peered":"2024-03-18T14:43:41.977174+0000","last_unstale":"2024-03-18T14:43:42.974151+0000","last_undegraded":"2024-03-18T14:43:42.974151+0000","last_fullsized":"2024-03-18T14:43:42.974151+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:33:19.352003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973721+0000","last_change":"2024-03-18T14:43:41.994593+0000","last_active":"2024-03-18T14:43:42.973721+0000","last_peered":"2024-03-18T14:43:42.973721+0000","last_clean":"2024-03-18T14:43:42.973721+0000","last_became_active":"2024-03-18T14:43:41.993184+0000","last_became_peered":"2024-03-18T14:43:41.993184+0000","last_unstale":"2024-03-18T14:43:42.973721+0000","last_undegraded":"2024-03-18T14:43:42.973721+0000","last_fullsized":"2024-03-18T14:43:42.973721+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:50.325632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971163+0000","last_change":"2024-03-18T14:43:41.994739+0000","last_active":"2024-03-18T14:43:42.971163+0000","last_peered":"2024-03-18T14:43:42.971163+0000","last_clean":"2024-03-18T14:43:42.971163+0000","last_became_active":"2024-03-18T14:43:41.994456+0000","last_became_peered":"2024-03-18T14:43:41.994456+0000","last_unstale":"2024-03-18T14:43:42.971163+0000","last_undegraded":"2024-03-18T14:43:42.971163+0000","last_fullsized":"2024-03-18T14:43:42.971163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:50.325632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973854+0000","last_change":"2024-03-18T14:43:41.986786+0000","last_active":"2024-03-18T14:43:42.973854+0000","last_peered":"2024-03-18T14:43:42.973854+0000","last_clean":"2024-03-18T14:43:42.973854+0000","last_became_active":"2024-03-18T14:43:41.986510+0000","last_became_peered":"2024-03-18T14:43:41.986510+0000","last_unstale":"2024-03-18T14:43:42.973854+0000","last_undegraded":"2024-03-18T14:43:42.973854+0000","last_fullsized":"2024-03-18T14:43:42.973854+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:02.354338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973784+0000","last_change":"2024-03-18T14:43:41.987134+0000","last_active":"2024-03-18T14:43:42.973784+0000","last_peered":"2024-03-18T14:43:42.973784+0000","last_clean":"2024-03-18T14:43:42.973784+0000","last_became_active":"2024-03-18T14:43:41.986850+0000","last_became_peered":"2024-03-18T14:43:41.986850+0000","last_unstale":"2024-03-18T14:43:42.973784+0000","last_undegraded":"2024-03-18T14:43:42.973784+0000","last_fullsized":"2024-03-18T14:43:42.973784+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:40:41.388142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974057+0000","last_change":"2024-03-18T14:43:41.982458+0000","last_active":"2024-03-18T14:43:42.974057+0000","last_peered":"2024-03-18T14:43:42.974057+0000","last_clean":"2024-03-18T14:43:42.974057+0000","last_became_active":"2024-03-18T14:43:41.982026+0000","last_became_peered":"2024-03-18T14:43:41.982026+0000","last_unstale":"2024-03-18T14:43:42.974057+0000","last_undegraded":"2024-03-18T14:43:42.974057+0000","last_fullsized":"2024-03-18T14:43:42.974057+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:46:10.236105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973880+0000","last_change":"2024-03-18T14:43:41.975787+0000","last_active":"2024-03-18T14:43:42.973880+0000","last_peered":"2024-03-18T14:43:42.973880+0000","last_clean":"2024-03-18T14:43:42.973880+0000","last_became_active":"2024-03-18T14:43:41.975477+0000","last_became_peered":"2024-03-18T14:43:41.975477+0000","last_unstale":"2024-03-18T14:43:42.973880+0000","last_undegraded":"2024-03-18T14:43:42.973880+0000","last_fullsized":"2024-03-18T14:43:42.973880+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:56.155609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970909+0000","last_change":"2024-03-18T14:43:42.000064+0000","last_active":"2024-03-18T14:43:42.970909+0000","last_peered":"2024-03-18T14:43:42.970909+0000","last_clean":"2024-03-18T14:43:42.970909+0000","last_became_active":"2024-03-18T14:43:41.999677+0000","last_became_peered":"2024-03-18T14:43:41.999677+0000","last_unstale":"2024-03-18T14:43:42.970909+0000","last_undegraded":"2024-03-18T14:43:42.970909+0000","last_fullsized":"2024-03-18T14:43:42.970909+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:38:56.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971875+0000","last_change":"2024-03-18T14:43:41.990616+0000","last_active":"2024-03-18T14:43:42.971875+0000","last_peered":"2024-03-18T14:43:42.971875+0000","last_clean":"2024-03-18T14:43:42.971875+0000","last_became_active":"2024-03-18T14:43:41.990303+0000","last_became_peered":"2024-03-18T14:43:41.990303+0000","last_unstale":"2024-03-18T14:43:42.971875+0000","last_undegraded":"2024-03-18T14:43:42.971875+0000","last_fullsized":"2024-03-18T14:43:42.971875+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:05.007512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971786+0000","last_change":"2024-03-18T14:43:40.983800+0000","last_active":"2024-03-18T14:43:42.971786+0000","last_peered":"2024-03-18T14:43:42.971786+0000","last_clean":"2024-03-18T14:43:42.971786+0000","last_became_active":"2024-03-18T14:43:40.983512+0000","last_became_peered":"2024-03-18T14:43:40.983512+0000","last_unstale":"2024-03-18T14:43:42.971786+0000","last_undegraded":"2024-03-18T14:43:42.971786+0000","last_fullsized":"2024-03-18T14:43:42.971786+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:39:45.521393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970408+0000","last_change":"2024-03-18T14:43:41.978941+0000","last_active":"2024-03-18T14:43:42.970408+0000","last_peered":"2024-03-18T14:43:42.970408+0000","last_clean":"2024-03-18T14:43:42.970408+0000","last_became_active":"2024-03-18T14:43:41.978681+0000","last_became_peered":"2024-03-18T14:43:41.978681+0000","last_unstale":"2024-03-18T14:43:42.970408+0000","last_undegraded":"2024-03-18T14:43:42.970408+0000","last_fullsized":"2024-03-18T14:43:42.970408+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T14:46:19.054657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973825+0000","last_change":"2024-03-18T14:43:40.986288+0000","last_active":"2024-03-18T14:43:42.973825+0000","last_peered":"2024-03-18T14:43:42.973825+0000","last_clean":"2024-03-18T14:43:42.973825+0000","last_became_active":"2024-03-18T14:43:40.985971+0000","last_became_peered":"2024-03-18T14:43:40.985971+0000","last_unstale":"2024-03-18T14:43:42.973825+0000","last_undegraded":"2024-03-18T14:43:42.973825+0000","last_fullsized":"2024-03-18T14:43:42.973825+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:17:00.912885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971479+0000","last_change":"2024-03-18T14:43:41.995874+0000","last_active":"2024-03-18T14:43:42.971479+0000","last_peered":"2024-03-18T14:43:42.971479+0000","last_clean":"2024-03-18T14:43:42.971479+0000","last_became_active":"2024-03-18T14:43:41.994701+0000","last_became_peered":"2024-03-18T14:43:41.994701+0000","last_unstale":"2024-03-18T14:43:42.971479+0000","last_undegraded":"2024-03-18T14:43:42.971479+0000","last_fullsized":"2024-03-18T14:43:42.971479+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:08.711617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971564+0000","last_change":"2024-03-18T14:43:40.983218+0000","last_active":"2024-03-18T14:43:42.971564+0000","last_peered":"2024-03-18T14:43:42.971564+0000","last_clean":"2024-03-18T14:43:42.971564+0000","last_became_active":"2024-03-18T14:43:40.982929+0000","last_became_peered":"2024-03-18T14:43:40.982929+0000","last_unstale":"2024-03-18T14:43:42.971564+0000","last_undegraded":"2024-03-18T14:43:42.971564+0000","last_fullsized":"2024-03-18T14:43:42.971564+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:30.323270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971301+0000","last_change":"2024-03-18T14:43:41.993827+0000","last_active":"2024-03-18T14:43:42.971301+0000","last_peered":"2024-03-18T14:43:42.971301+0000","last_clean":"2024-03-18T14:43:42.971301+0000","last_became_active":"2024-03-18T14:43:41.993460+0000","last_became_peered":"2024-03-18T14:43:41.993460+0000","last_unstale":"2024-03-18T14:43:42.971301+0000","last_undegraded":"2024-03-18T14:43:42.971301+0000","last_fullsized":"2024-03-18T14:43:42.971301+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:56.329557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974587+0000","last_change":"2024-03-18T14:43:40.986774+0000","last_active":"2024-03-18T14:43:42.974587+0000","last_peered":"2024-03-18T14:43:42.974587+0000","last_clean":"2024-03-18T14:43:42.974587+0000","last_became_active":"2024-03-18T14:43:40.986438+0000","last_became_peered":"2024-03-18T14:43:40.986438+0000","last_unstale":"2024-03-18T14:43:42.974587+0000","last_undegraded":"2024-03-18T14:43:42.974587+0000","last_fullsized":"2024-03-18T14:43:42.974587+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:58.935947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973475+0000","last_change":"2024-03-18T14:43:41.976102+0000","last_active":"2024-03-18T14:43:42.973475+0000","last_peered":"2024-03-18T14:43:42.973475+0000","last_clean":"2024-03-18T14:43:42.973475+0000","last_became_active":"2024-03-18T14:43:41.975779+0000","last_became_peered":"2024-03-18T14:43:41.975779+0000","last_unstale":"2024-03-18T14:43:42.973475+0000","last_undegraded":"2024-03-18T14:43:42.973475+0000","last_fullsized":"2024-03-18T14:43:42.973475+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:13.535288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974765+0000","last_change":"2024-03-18T14:43:40.986248+0000","last_active":"2024-03-18T14:43:42.974765+0000","last_peered":"2024-03-18T14:43:42.974765+0000","last_clean":"2024-03-18T14:43:42.974765+0000","last_became_active":"2024-03-18T14:43:40.985994+0000","last_became_peered":"2024-03-18T14:43:40.985994+0000","last_unstale":"2024-03-18T14:43:42.974765+0000","last_undegraded":"2024-03-18T14:43:42.974765+0000","last_fullsized":"2024-03-18T14:43:42.974765+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:50.895017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973530+0000","last_change":"2024-03-18T14:43:41.990665+0000","last_active":"2024-03-18T14:43:42.973530+0000","last_peered":"2024-03-18T14:43:42.973530+0000","last_clean":"2024-03-18T14:43:42.973530+0000","last_became_active":"2024-03-18T14:43:41.990346+0000","last_became_peered":"2024-03-18T14:43:41.990346+0000","last_unstale":"2024-03-18T14:43:42.973530+0000","last_undegraded":"2024-03-18T14:43:42.973530+0000","last_fullsized":"2024-03-18T14:43:42.973530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:18.541018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971144+0000","last_change":"2024-03-18T14:43:40.983138+0000","last_active":"2024-03-18T14:43:42.971144+0000","last_peered":"2024-03-18T14:43:42.971144+0000","last_clean":"2024-03-18T14:43:42.971144+0000","last_became_active":"2024-03-18T14:43:40.982801+0000","last_became_peered":"2024-03-18T14:43:40.982801+0000","last_unstale":"2024-03-18T14:43:42.971144+0000","last_undegraded":"2024-03-18T14:43:42.971144+0000","last_fullsized":"2024-03-18T14:43:42.971144+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:20.609968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971359+0000","last_change":"2024-03-18T14:43:41.990846+0000","last_active":"2024-03-18T14:43:42.971359+0000","last_peered":"2024-03-18T14:43:42.971359+0000","last_clean":"2024-03-18T14:43:42.971359+0000","last_became_active":"2024-03-18T14:43:41.990552+0000","last_became_peered":"2024-03-18T14:43:41.990552+0000","last_unstale":"2024-03-18T14:43:42.971359+0000","last_undegraded":"2024-03-18T14:43:42.971359+0000","last_fullsized":"2024-03-18T14:43:42.971359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:33.841740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973702+0000","last_change":"2024-03-18T14:43:40.979927+0000","last_active":"2024-03-18T14:43:42.973702+0000","last_peered":"2024-03-18T14:43:42.973702+0000","last_clean":"2024-03-18T14:43:42.973702+0000","last_became_active":"2024-03-18T14:43:40.979561+0000","last_became_peered":"2024-03-18T14:43:40.979561+0000","last_unstale":"2024-03-18T14:43:42.973702+0000","last_undegraded":"2024-03-18T14:43:42.973702+0000","last_fullsized":"2024-03-18T14:43:42.973702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:26.204701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973360+0000","last_change":"2024-03-18T14:43:41.989410+0000","last_active":"2024-03-18T14:43:42.973360+0000","last_peered":"2024-03-18T14:43:42.973360+0000","last_clean":"2024-03-18T14:43:42.973360+0000","last_became_active":"2024-03-18T14:43:41.988707+0000","last_became_peered":"2024-03-18T14:43:41.988707+0000","last_unstale":"2024-03-18T14:43:42.973360+0000","last_undegraded":"2024-03-18T14:43:42.973360+0000","last_fullsized":"2024-03-18T14:43:42.973360+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:41.135260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974745+0000","last_change":"2024-03-18T14:43:40.988370+0000","last_active":"2024-03-18T14:43:42.974745+0000","last_peered":"2024-03-18T14:43:42.974745+0000","last_clean":"2024-03-18T14:43:42.974745+0000","last_became_active":"2024-03-18T14:43:40.988143+0000","last_became_peered":"2024-03-18T14:43:40.988143+0000","last_unstale":"2024-03-18T14:43:42.974745+0000","last_undegraded":"2024-03-18T14:43:42.974745+0000","last_fullsized":"2024-03-18T14:43:42.974745+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:19.741331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975115+0000","last_change":"2024-03-18T14:43:41.992354+0000","last_active":"2024-03-18T14:43:42.975115+0000","last_peered":"2024-03-18T14:43:42.975115+0000","last_clean":"2024-03-18T14:43:42.975115+0000","last_became_active":"2024-03-18T14:43:41.991543+0000","last_became_peered":"2024-03-18T14:43:41.991543+0000","last_unstale":"2024-03-18T14:43:42.975115+0000","last_undegraded":"2024-03-18T14:43:42.975115+0000","last_fullsized":"2024-03-18T14:43:42.975115+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:21.550317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974276+0000","last_change":"2024-03-18T14:43:40.990848+0000","last_active":"2024-03-18T14:43:42.974276+0000","last_peered":"2024-03-18T14:43:42.974276+0000","last_clean":"2024-03-18T14:43:42.974276+0000","last_became_active":"2024-03-18T14:43:40.990506+0000","last_became_peered":"2024-03-18T14:43:40.990506+0000","last_unstale":"2024-03-18T14:43:42.974276+0000","last_undegraded":"2024-03-18T14:43:42.974276+0000","last_fullsized":"2024-03-18T14:43:42.974276+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:18:23.568861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974309+0000","last_change":"2024-03-18T14:43:41.982880+0000","last_active":"2024-03-18T14:43:42.974309+0000","last_peered":"2024-03-18T14:43:42.974309+0000","last_clean":"2024-03-18T14:43:42.974309+0000","last_became_active":"2024-03-18T14:43:41.982614+0000","last_became_peered":"2024-03-18T14:43:41.982614+0000","last_unstale":"2024-03-18T14:43:42.974309+0000","last_undegraded":"2024-03-18T14:43:42.974309+0000","last_fullsized":"2024-03-18T14:43:42.974309+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:15.827363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973837+0000","last_change":"2024-03-18T14:43:40.984498+0000","last_active":"2024-03-18T14:43:42.973837+0000","last_peered":"2024-03-18T14:43:42.973837+0000","last_clean":"2024-03-18T14:43:42.973837+0000","last_became_active":"2024-03-18T14:43:40.984005+0000","last_became_peered":"2024-03-18T14:43:40.984005+0000","last_unstale":"2024-03-18T14:43:42.973837+0000","last_undegraded":"2024-03-18T14:43:42.973837+0000","last_fullsized":"2024-03-18T14:43:42.973837+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:31:46.300184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970137+0000","last_change":"2024-03-18T14:43:41.987804+0000","last_active":"2024-03-18T14:43:42.970137+0000","last_peered":"2024-03-18T14:43:42.970137+0000","last_clean":"2024-03-18T14:43:42.970137+0000","last_became_active":"2024-03-18T14:43:41.987502+0000","last_became_peered":"2024-03-18T14:43:41.987502+0000","last_unstale":"2024-03-18T14:43:42.970137+0000","last_undegraded":"2024-03-18T14:43:42.970137+0000","last_fullsized":"2024-03-18T14:43:42.970137+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.004568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974399+0000","last_change":"2024-03-18T14:43:40.990752+0000","last_active":"2024-03-18T14:43:42.974399+0000","last_peered":"2024-03-18T14:43:42.974399+0000","last_clean":"2024-03-18T14:43:42.974399+0000","last_became_active":"2024-03-18T14:43:40.990438+0000","last_became_peered":"2024-03-18T14:43:40.990438+0000","last_unstale":"2024-03-18T14:43:42.974399+0000","last_undegraded":"2024-03-18T14:43:42.974399+0000","last_fullsized":"2024-03-18T14:43:42.974399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:20.609968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974336+0000","last_change":"2024-03-18T14:43:41.993639+0000","last_active":"2024-03-18T14:43:42.974336+0000","last_peered":"2024-03-18T14:43:42.974336+0000","last_clean":"2024-03-18T14:43:42.974336+0000","last_became_active":"2024-03-18T14:43:41.993353+0000","last_became_peered":"2024-03-18T14:43:41.993353+0000","last_unstale":"2024-03-18T14:43:42.974336+0000","last_undegraded":"2024-03-18T14:43:42.974336+0000","last_fullsized":"2024-03-18T14:43:42.974336+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:56.243015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975063+0000","last_change":"2024-03-18T14:43:40.983867+0000","last_active":"2024-03-18T14:43:42.975063+0000","last_peered":"2024-03-18T14:43:42.975063+0000","last_clean":"2024-03-18T14:43:42.975063+0000","last_became_active":"2024-03-18T14:43:40.983577+0000","last_became_peered":"2024-03-18T14:43:40.983577+0000","last_unstale":"2024-03-18T14:43:42.975063+0000","last_undegraded":"2024-03-18T14:43:42.975063+0000","last_fullsized":"2024-03-18T14:43:42.975063+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:45.865942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973430+0000","last_change":"2024-03-18T14:43:41.999442+0000","last_active":"2024-03-18T14:43:42.973430+0000","last_peered":"2024-03-18T14:43:42.973430+0000","last_clean":"2024-03-18T14:43:42.973430+0000","last_became_active":"2024-03-18T14:43:41.999197+0000","last_became_peered":"2024-03-18T14:43:41.999197+0000","last_unstale":"2024-03-18T14:43:42.973430+0000","last_undegraded":"2024-03-18T14:43:42.973430+0000","last_fullsized":"2024-03-18T14:43:42.973430+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:38:56.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973637+0000","last_change":"2024-03-18T14:43:40.987950+0000","last_active":"2024-03-18T14:43:42.973637+0000","last_peered":"2024-03-18T14:43:42.973637+0000","last_clean":"2024-03-18T14:43:42.973637+0000","last_became_active":"2024-03-18T14:43:40.987677+0000","last_became_peered":"2024-03-18T14:43:40.987677+0000","last_unstale":"2024-03-18T14:43:42.973637+0000","last_undegraded":"2024-03-18T14:43:42.973637+0000","last_fullsized":"2024-03-18T14:43:42.973637+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:33.490428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974172+0000","last_change":"2024-03-18T14:43:41.986125+0000","last_active":"2024-03-18T14:43:42.974172+0000","last_peered":"2024-03-18T14:43:42.974172+0000","last_clean":"2024-03-18T14:43:42.974172+0000","last_became_active":"2024-03-18T14:43:41.985878+0000","last_became_peered":"2024-03-18T14:43:41.985878+0000","last_unstale":"2024-03-18T14:43:42.974172+0000","last_undegraded":"2024-03-18T14:43:42.974172+0000","last_fullsized":"2024-03-18T14:43:42.974172+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:16.536785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971072+0000","last_change":"2024-03-18T14:43:40.985906+0000","last_active":"2024-03-18T14:43:42.971072+0000","last_peered":"2024-03-18T14:43:42.971072+0000","last_clean":"2024-03-18T14:43:42.971072+0000","last_became_active":"2024-03-18T14:43:40.985591+0000","last_became_peered":"2024-03-18T14:43:40.985591+0000","last_unstale":"2024-03-18T14:43:42.971072+0000","last_undegraded":"2024-03-18T14:43:42.971072+0000","last_fullsized":"2024-03-18T14:43:42.971072+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:27.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974742+0000","last_change":"2024-03-18T14:43:41.984089+0000","last_active":"2024-03-18T14:43:42.974742+0000","last_peered":"2024-03-18T14:43:42.974742+0000","last_clean":"2024-03-18T14:43:42.974742+0000","last_became_active":"2024-03-18T14:43:41.983697+0000","last_became_peered":"2024-03-18T14:43:41.983697+0000","last_unstale":"2024-03-18T14:43:42.974742+0000","last_undegraded":"2024-03-18T14:43:42.974742+0000","last_fullsized":"2024-03-18T14:43:42.974742+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:18:24.577769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973801+0000","last_change":"2024-03-18T14:43:40.981949+0000","last_active":"2024-03-18T14:43:42.973801+0000","last_peered":"2024-03-18T14:43:42.973801+0000","last_clean":"2024-03-18T14:43:42.973801+0000","last_became_active":"2024-03-18T14:43:40.981625+0000","last_became_peered":"2024-03-18T14:43:40.981625+0000","last_unstale":"2024-03-18T14:43:42.973801+0000","last_undegraded":"2024-03-18T14:43:42.973801+0000","last_fullsized":"2024-03-18T14:43:42.973801+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:39:45.521393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974170+0000","last_change":"2024-03-18T14:43:41.989933+0000","last_active":"2024-03-18T14:43:42.974170+0000","last_peered":"2024-03-18T14:43:42.974170+0000","last_clean":"2024-03-18T14:43:42.974170+0000","last_became_active":"2024-03-18T14:43:41.988547+0000","last_became_peered":"2024-03-18T14:43:41.988547+0000","last_unstale":"2024-03-18T14:43:42.974170+0000","last_undegraded":"2024-03-18T14:43:42.974170+0000","last_fullsized":"2024-03-18T14:43:42.974170+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:39.379173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973696+0000","last_change":"2024-03-18T14:43:40.991385+0000","last_active":"2024-03-18T14:43:42.973696+0000","last_peered":"2024-03-18T14:43:42.973696+0000","last_clean":"2024-03-18T14:43:42.973696+0000","last_became_active":"2024-03-18T14:43:40.991010+0000","last_became_peered":"2024-03-18T14:43:40.991010+0000","last_unstale":"2024-03-18T14:43:42.973696+0000","last_undegraded":"2024-03-18T14:43:42.973696+0000","last_fullsized":"2024-03-18T14:43:42.973696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:28:29.280355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974175+0000","last_change":"2024-03-18T14:43:41.993861+0000","last_active":"2024-03-18T14:43:42.974175+0000","last_peered":"2024-03-18T14:43:42.974175+0000","last_clean":"2024-03-18T14:43:42.974175+0000","last_became_active":"2024-03-18T14:43:41.993590+0000","last_became_peered":"2024-03-18T14:43:41.993590+0000","last_unstale":"2024-03-18T14:43:42.974175+0000","last_undegraded":"2024-03-18T14:43:42.974175+0000","last_fullsized":"2024-03-18T14:43:42.974175+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:38:56.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974121+0000","last_change":"2024-03-18T14:43:40.982412+0000","last_active":"2024-03-18T14:43:42.974121+0000","last_peered":"2024-03-18T14:43:42.974121+0000","last_clean":"2024-03-18T14:43:42.974121+0000","last_became_active":"2024-03-18T14:43:40.982153+0000","last_became_peered":"2024-03-18T14:43:40.982153+0000","last_unstale":"2024-03-18T14:43:42.974121+0000","last_undegraded":"2024-03-18T14:43:42.974121+0000","last_fullsized":"2024-03-18T14:43:42.974121+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:44.651178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973809+0000","last_change":"2024-03-18T14:43:41.998734+0000","last_active":"2024-03-18T14:43:42.973809+0000","last_peered":"2024-03-18T14:43:42.973809+0000","last_clean":"2024-03-18T14:43:42.973809+0000","last_became_active":"2024-03-18T14:43:41.998445+0000","last_became_peered":"2024-03-18T14:43:41.998445+0000","last_unstale":"2024-03-18T14:43:42.973809+0000","last_undegraded":"2024-03-18T14:43:42.973809+0000","last_fullsized":"2024-03-18T14:43:42.973809+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:15.827363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971724+0000","last_change":"2024-03-18T14:43:40.986458+0000","last_active":"2024-03-18T14:43:42.971724+0000","last_peered":"2024-03-18T14:43:42.971724+0000","last_clean":"2024-03-18T14:43:42.971724+0000","last_became_active":"2024-03-18T14:43:40.986161+0000","last_became_peered":"2024-03-18T14:43:40.986161+0000","last_unstale":"2024-03-18T14:43:42.971724+0000","last_undegraded":"2024-03-18T14:43:42.971724+0000","last_fullsized":"2024-03-18T14:43:42.971724+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:20:46.331310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973754+0000","last_change":"2024-03-18T14:43:41.988040+0000","last_active":"2024-03-18T14:43:42.973754+0000","last_peered":"2024-03-18T14:43:42.973754+0000","last_clean":"2024-03-18T14:43:42.973754+0000","last_became_active":"2024-03-18T14:43:41.987758+0000","last_became_peered":"2024-03-18T14:43:41.987758+0000","last_unstale":"2024-03-18T14:43:42.973754+0000","last_undegraded":"2024-03-18T14:43:42.973754+0000","last_fullsized":"2024-03-18T14:43:42.973754+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:13.115507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973614+0000","last_change":"2024-03-18T14:43:40.986938+0000","last_active":"2024-03-18T14:43:42.973614+0000","last_peered":"2024-03-18T14:43:42.973614+0000","last_clean":"2024-03-18T14:43:42.973614+0000","last_became_active":"2024-03-18T14:43:40.986666+0000","last_became_peered":"2024-03-18T14:43:40.986666+0000","last_unstale":"2024-03-18T14:43:42.973614+0000","last_undegraded":"2024-03-18T14:43:42.973614+0000","last_fullsized":"2024-03-18T14:43:42.973614+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:33:22.924944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974446+0000","last_change":"2024-03-18T14:43:41.993435+0000","last_active":"2024-03-18T14:43:42.974446+0000","last_peered":"2024-03-18T14:43:42.974446+0000","last_clean":"2024-03-18T14:43:42.974446+0000","last_became_active":"2024-03-18T14:43:41.993185+0000","last_became_peered":"2024-03-18T14:43:41.993185+0000","last_unstale":"2024-03-18T14:43:42.974446+0000","last_undegraded":"2024-03-18T14:43:42.974446+0000","last_fullsized":"2024-03-18T14:43:42.974446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:16:08.711617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971574+0000","last_change":"2024-03-18T14:43:40.989351+0000","last_active":"2024-03-18T14:43:42.971574+0000","last_peered":"2024-03-18T14:43:42.971574+0000","last_clean":"2024-03-18T14:43:42.971574+0000","last_became_active":"2024-03-18T14:43:40.989054+0000","last_became_peered":"2024-03-18T14:43:40.989054+0000","last_unstale":"2024-03-18T14:43:42.971574+0000","last_undegraded":"2024-03-18T14:43:42.971574+0000","last_fullsized":"2024-03-18T14:43:42.971574+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:59.144538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973482+0000","last_change":"2024-03-18T14:43:41.994745+0000","last_active":"2024-03-18T14:43:42.973482+0000","last_peered":"2024-03-18T14:43:42.973482+0000","last_clean":"2024-03-18T14:43:42.973482+0000","last_became_active":"2024-03-18T14:43:41.993838+0000","last_became_peered":"2024-03-18T14:43:41.993838+0000","last_unstale":"2024-03-18T14:43:42.973482+0000","last_undegraded":"2024-03-18T14:43:42.973482+0000","last_fullsized":"2024-03-18T14:43:42.973482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:16.536785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973886+0000","last_change":"2024-03-18T14:43:40.982844+0000","last_active":"2024-03-18T14:43:42.973886+0000","last_peered":"2024-03-18T14:43:42.973886+0000","last_clean":"2024-03-18T14:43:42.973886+0000","last_became_active":"2024-03-18T14:43:40.982535+0000","last_became_peered":"2024-03-18T14:43:40.982535+0000","last_unstale":"2024-03-18T14:43:42.973886+0000","last_undegraded":"2024-03-18T14:43:42.973886+0000","last_fullsized":"2024-03-18T14:43:42.973886+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:49:51.094883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971320+0000","last_change":"2024-03-18T14:43:41.994275+0000","last_active":"2024-03-18T14:43:42.971320+0000","last_peered":"2024-03-18T14:43:42.971320+0000","last_clean":"2024-03-18T14:43:42.971320+0000","last_became_active":"2024-03-18T14:43:41.993874+0000","last_became_peered":"2024-03-18T14:43:41.993874+0000","last_unstale":"2024-03-18T14:43:42.971320+0000","last_undegraded":"2024-03-18T14:43:42.971320+0000","last_fullsized":"2024-03-18T14:43:42.971320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:56.243015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970214+0000","last_change":"2024-03-18T14:43:40.982113+0000","last_active":"2024-03-18T14:43:42.970214+0000","last_peered":"2024-03-18T14:43:42.970214+0000","last_clean":"2024-03-18T14:43:42.970214+0000","last_became_active":"2024-03-18T14:43:40.981369+0000","last_became_peered":"2024-03-18T14:43:40.981369+0000","last_unstale":"2024-03-18T14:43:42.970214+0000","last_undegraded":"2024-03-18T14:43:42.970214+0000","last_fullsized":"2024-03-18T14:43:42.970214+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:53.144792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970926+0000","last_change":"2024-03-18T14:43:41.993255+0000","last_active":"2024-03-18T14:43:42.970926+0000","last_peered":"2024-03-18T14:43:42.970926+0000","last_clean":"2024-03-18T14:43:42.970926+0000","last_became_active":"2024-03-18T14:43:41.992976+0000","last_became_peered":"2024-03-18T14:43:41.992976+0000","last_unstale":"2024-03-18T14:43:42.970926+0000","last_undegraded":"2024-03-18T14:43:42.970926+0000","last_fullsized":"2024-03-18T14:43:42.970926+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:16.536785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973541+0000","last_change":"2024-03-18T14:43:40.989232+0000","last_active":"2024-03-18T14:43:42.973541+0000","last_peered":"2024-03-18T14:43:42.973541+0000","last_clean":"2024-03-18T14:43:42.973541+0000","last_became_active":"2024-03-18T14:43:40.988982+0000","last_became_peered":"2024-03-18T14:43:40.988982+0000","last_unstale":"2024-03-18T14:43:42.973541+0000","last_undegraded":"2024-03-18T14:43:42.973541+0000","last_fullsized":"2024-03-18T14:43:42.973541+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:20.541410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973864+0000","last_change":"2024-03-18T14:43:41.987124+0000","last_active":"2024-03-18T14:43:42.973864+0000","last_peered":"2024-03-18T14:43:42.973864+0000","last_clean":"2024-03-18T14:43:42.973864+0000","last_became_active":"2024-03-18T14:43:41.986518+0000","last_became_peered":"2024-03-18T14:43:41.986518+0000","last_unstale":"2024-03-18T14:43:42.973864+0000","last_undegraded":"2024-03-18T14:43:42.973864+0000","last_fullsized":"2024-03-18T14:43:42.973864+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:56.329557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970407+0000","last_change":"2024-03-18T14:43:40.988468+0000","last_active":"2024-03-18T14:43:42.970407+0000","last_peered":"2024-03-18T14:43:42.970407+0000","last_clean":"2024-03-18T14:43:42.970407+0000","last_became_active":"2024-03-18T14:43:40.988130+0000","last_became_peered":"2024-03-18T14:43:40.988130+0000","last_unstale":"2024-03-18T14:43:42.970407+0000","last_undegraded":"2024-03-18T14:43:42.970407+0000","last_fullsized":"2024-03-18T14:43:42.970407+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:27.223941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973978+0000","last_change":"2024-03-18T14:43:41.990918+0000","last_active":"2024-03-18T14:43:42.973978+0000","last_peered":"2024-03-18T14:43:42.973978+0000","last_clean":"2024-03-18T14:43:42.973978+0000","last_became_active":"2024-03-18T14:43:41.990598+0000","last_became_peered":"2024-03-18T14:43:41.990598+0000","last_unstale":"2024-03-18T14:43:42.973978+0000","last_undegraded":"2024-03-18T14:43:42.973978+0000","last_fullsized":"2024-03-18T14:43:42.973978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:56.243015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.986543+0000","last_change":"2024-03-18T14:43:40.984516+0000","last_active":"2024-03-18T14:43:42.986543+0000","last_peered":"2024-03-18T14:43:42.986543+0000","last_clean":"2024-03-18T14:43:42.986543+0000","last_became_active":"2024-03-18T14:43:40.984037+0000","last_became_peered":"2024-03-18T14:43:40.984037+0000","last_unstale":"2024-03-18T14:43:42.986543+0000","last_undegraded":"2024-03-18T14:43:42.986543+0000","last_fullsized":"2024-03-18T14:43:42.986543+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:20.541410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.970269+0000","last_change":"2024-03-18T14:43:41.984502+0000","last_active":"2024-03-18T14:43:42.970269+0000","last_peered":"2024-03-18T14:43:42.970269+0000","last_clean":"2024-03-18T14:43:42.970269+0000","last_became_active":"2024-03-18T14:43:41.984186+0000","last_became_peered":"2024-03-18T14:43:41.984186+0000","last_unstale":"2024-03-18T14:43:42.970269+0000","last_undegraded":"2024-03-18T14:43:42.970269+0000","last_fullsized":"2024-03-18T14:43:42.970269+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:16.791852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971307+0000","last_change":"2024-03-18T14:43:40.979968+0000","last_active":"2024-03-18T14:43:42.971307+0000","last_peered":"2024-03-18T14:43:42.971307+0000","last_clean":"2024-03-18T14:43:42.971307+0000","last_became_active":"2024-03-18T14:43:40.979691+0000","last_became_peered":"2024-03-18T14:43:40.979691+0000","last_unstale":"2024-03-18T14:43:42.971307+0000","last_undegraded":"2024-03-18T14:43:42.971307+0000","last_fullsized":"2024-03-18T14:43:42.971307+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:36:26.213692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971641+0000","last_change":"2024-03-18T14:43:41.989816+0000","last_active":"2024-03-18T14:43:42.971641+0000","last_peered":"2024-03-18T14:43:42.971641+0000","last_clean":"2024-03-18T14:43:42.971641+0000","last_became_active":"2024-03-18T14:43:41.989471+0000","last_became_peered":"2024-03-18T14:43:41.989471+0000","last_unstale":"2024-03-18T14:43:42.971641+0000","last_undegraded":"2024-03-18T14:43:42.971641+0000","last_fullsized":"2024-03-18T14:43:42.971641+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:22.376659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974585+0000","last_change":"2024-03-18T14:43:40.982991+0000","last_active":"2024-03-18T14:43:42.974585+0000","last_peered":"2024-03-18T14:43:42.974585+0000","last_clean":"2024-03-18T14:43:42.974585+0000","last_became_active":"2024-03-18T14:43:40.982709+0000","last_became_peered":"2024-03-18T14:43:40.982709+0000","last_unstale":"2024-03-18T14:43:42.974585+0000","last_undegraded":"2024-03-18T14:43:42.974585+0000","last_fullsized":"2024-03-18T14:43:42.974585+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:57.222848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973617+0000","last_change":"2024-03-18T14:43:41.976076+0000","last_active":"2024-03-18T14:43:42.973617+0000","last_peered":"2024-03-18T14:43:42.973617+0000","last_clean":"2024-03-18T14:43:42.973617+0000","last_became_active":"2024-03-18T14:43:41.975418+0000","last_became_peered":"2024-03-18T14:43:41.975418+0000","last_unstale":"2024-03-18T14:43:42.973617+0000","last_undegraded":"2024-03-18T14:43:42.973617+0000","last_fullsized":"2024-03-18T14:43:42.973617+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:37:54.128119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.969987+0000","last_change":"2024-03-18T14:43:40.979481+0000","last_active":"2024-03-18T14:43:42.969987+0000","last_peered":"2024-03-18T14:43:42.969987+0000","last_clean":"2024-03-18T14:43:42.969987+0000","last_became_active":"2024-03-18T14:43:40.979205+0000","last_became_peered":"2024-03-18T14:43:40.979205+0000","last_unstale":"2024-03-18T14:43:42.969987+0000","last_undegraded":"2024-03-18T14:43:42.969987+0000","last_fullsized":"2024-03-18T14:43:42.969987+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:46.392263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974059+0000","last_change":"2024-03-18T14:43:41.983354+0000","last_active":"2024-03-18T14:43:42.974059+0000","last_peered":"2024-03-18T14:43:42.974059+0000","last_clean":"2024-03-18T14:43:42.974059+0000","last_became_active":"2024-03-18T14:43:41.983072+0000","last_became_peered":"2024-03-18T14:43:41.983072+0000","last_unstale":"2024-03-18T14:43:42.974059+0000","last_undegraded":"2024-03-18T14:43:42.974059+0000","last_fullsized":"2024-03-18T14:43:42.974059+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:26:47.695549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.975063+0000","last_change":"2024-03-18T14:43:40.984587+0000","last_active":"2024-03-18T14:43:42.975063+0000","last_peered":"2024-03-18T14:43:42.975063+0000","last_clean":"2024-03-18T14:43:42.975063+0000","last_became_active":"2024-03-18T14:43:40.984301+0000","last_became_peered":"2024-03-18T14:43:40.984301+0000","last_unstale":"2024-03-18T14:43:42.975063+0000","last_undegraded":"2024-03-18T14:43:42.975063+0000","last_fullsized":"2024-03-18T14:43:42.975063+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:59.274683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.969891+0000","last_change":"2024-03-18T14:43:41.986909+0000","last_active":"2024-03-18T14:43:42.969891+0000","last_peered":"2024-03-18T14:43:42.969891+0000","last_clean":"2024-03-18T14:43:42.969891+0000","last_became_active":"2024-03-18T14:43:41.986636+0000","last_became_peered":"2024-03-18T14:43:41.986636+0000","last_unstale":"2024-03-18T14:43:42.969891+0000","last_undegraded":"2024-03-18T14:43:42.969891+0000","last_fullsized":"2024-03-18T14:43:42.969891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:33.841740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.969997+0000","last_change":"2024-03-18T14:43:40.978243+0000","last_active":"2024-03-18T14:43:42.969997+0000","last_peered":"2024-03-18T14:43:42.969997+0000","last_clean":"2024-03-18T14:43:42.969997+0000","last_became_active":"2024-03-18T14:43:40.977980+0000","last_became_peered":"2024-03-18T14:43:40.977980+0000","last_unstale":"2024-03-18T14:43:42.969997+0000","last_undegraded":"2024-03-18T14:43:42.969997+0000","last_fullsized":"2024-03-18T14:43:42.969997+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:02.875572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973705+0000","last_change":"2024-03-18T14:43:41.988993+0000","last_active":"2024-03-18T14:43:42.973705+0000","last_peered":"2024-03-18T14:43:42.973705+0000","last_clean":"2024-03-18T14:43:42.973705+0000","last_became_active":"2024-03-18T14:43:41.988308+0000","last_became_peered":"2024-03-18T14:43:41.988308+0000","last_unstale":"2024-03-18T14:43:42.973705+0000","last_undegraded":"2024-03-18T14:43:42.973705+0000","last_fullsized":"2024-03-18T14:43:42.973705+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:24.113873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973396+0000","last_change":"2024-03-18T14:43:40.979654+0000","last_active":"2024-03-18T14:43:42.973396+0000","last_peered":"2024-03-18T14:43:42.973396+0000","last_clean":"2024-03-18T14:43:42.973396+0000","last_became_active":"2024-03-18T14:43:40.979137+0000","last_became_peered":"2024-03-18T14:43:40.979137+0000","last_unstale":"2024-03-18T14:43:42.973396+0000","last_undegraded":"2024-03-18T14:43:42.973396+0000","last_fullsized":"2024-03-18T14:43:42.973396+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:46:09.227198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.974070+0000","last_change":"2024-03-18T14:43:41.986366+0000","last_active":"2024-03-18T14:43:42.974070+0000","last_peered":"2024-03-18T14:43:42.974070+0000","last_clean":"2024-03-18T14:43:42.974070+0000","last_became_active":"2024-03-18T14:43:41.986082+0000","last_became_peered":"2024-03-18T14:43:41.986082+0000","last_unstale":"2024-03-18T14:43:42.974070+0000","last_undegraded":"2024-03-18T14:43:42.974070+0000","last_fullsized":"2024-03-18T14:43:42.974070+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:40.957466+0000","last_clean_scrub_stamp":"2024-03-18T14:43:40.957466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:14:46.090742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.971131+0000","last_change":"2024-03-18T14:43:40.978079+0000","last_active":"2024-03-18T14:43:42.971131+0000","last_peered":"2024-03-18T14:43:42.971131+0000","last_clean":"2024-03-18T14:43:42.971131+0000","last_became_active":"2024-03-18T14:43:40.977784+0000","last_became_peered":"2024-03-18T14:43:40.977784+0000","last_unstale":"2024-03-18T14:43:42.971131+0000","last_undegraded":"2024-03-18T14:43:42.971131+0000","last_fullsized":"2024-03-18T14:43:42.971131+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:43:39.948558+0000","last_clean_scrub_stamp":"2024-03-18T14:43:39.948558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:53.144792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":233,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973747+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:43:42.973747+0000","last_peered":"2024-03-18T14:43:42.973747+0000","last_clean":"2024-03-18T14:43:42.973747+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:43:42.973747+0000","last_undegraded":"2024-03-18T14:43:42.973747+0000","last_fullsized":"2024-03-18T14:43:42.973747+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":297457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":894677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":305761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":667648,"data_stored":297513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":663552,"data_stored":297547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1269760,"data_stored":890019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:43:46.419 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T14:43:46.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T14:43:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.593+0000 7f709739d640 1 Processor -- start 2024-03-18T14:43:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.593+0000 7f709739d640 1 -- start start 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f709739d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f709015dd50 0x7f70900b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f709739d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70900b5cd0 0x7f70900ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f709739d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7090161bc0 con 0x7f709015e130 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f709739d640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7090161ea0 con 0x7f70900b5cd0 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f709739d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7090162180 con 0x7f709015dd50 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7096b9c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70900b5cd0 0x7f70900ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7095b9a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f709015e130 0x7f70900b55c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58092/0 (socket says 172.21.15.6:58092) 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7095b9a640 1 -- 172.21.15.6:0/2662603866 learned_addr learned my addr 172.21.15.6:0/2662603866 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f709639b640 1 --2- 172.21.15.6:0/2662603866 >> v2:172.21.15.92:3300/0 conn(0x7f709015dd50 0x7f70900b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7096b9c640 1 -- 172.21.15.6:0/2662603866 >> v2:172.21.15.92:3300/0 conn(0x7f709015dd50 msgr2=0x7f70900b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7096b9c640 1 --2- 172.21.15.6:0/2662603866 >> v2:172.21.15.92:3300/0 conn(0x7f709015dd50 0x7f70900b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7096b9c640 1 -- 172.21.15.6:0/2662603866 >> v1:172.21.15.6:6789/0 conn(0x7f709015e130 legacy=0x7f70900b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7096b9c640 1 -- 172.21.15.6:0/2662603866 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7090162460 con 0x7f70900b5cd0 2024-03-18T14:43:46.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.594+0000 7f7095399640 1 -- 172.21.15.6:0/2662603866 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910788446 0 0) 0x7f7090161bc0 con 0x7f709015e130 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.595+0000 7f7096b9c640 1 --2- 172.21.15.6:0/2662603866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70900b5cd0 0x7f70900ce3a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f70900738d0 tx=0x7f708802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38a9ddf14de28f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.595+0000 7f7095399640 1 -- 172.21.15.6:0/2662603866 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088039070 con 0x7f70900b5cd0 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.595+0000 7f7095399640 1 -- 172.21.15.6:0/2662603866 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708802cc80 con 0x7f70900b5cd0 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.595+0000 7f7095399640 1 -- 172.21.15.6:0/2662603866 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088034660 con 0x7f70900b5cd0 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 -- 172.21.15.6:0/2662603866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70900b5cd0 msgr2=0x7f70900ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 --2- 172.21.15.6:0/2662603866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70900b5cd0 0x7f70900ce3a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f70900738d0 tx=0x7f708802bed0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 -- 172.21.15.6:0/2662603866 shutdown_connections 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 --2- 172.21.15.6:0/2662603866 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70900b5cd0 0x7f70900ce3a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 --2- 172.21.15.6:0/2662603866 >> v2:172.21.15.92:3300/0 conn(0x7f709015dd50 0x7f70900b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 -- 172.21.15.6:0/2662603866 >> 172.21.15.6:0/2662603866 conn(0x7f70900d05c0 msgr2=0x7f70900d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 -- 172.21.15.6:0/2662603866 shutdown_connections 2024-03-18T14:43:46.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.596+0000 7f709739d640 1 -- 172.21.15.6:0/2662603866 wait complete. 2024-03-18T14:43:46.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.597+0000 7f709739d640 1 Processor -- start 2024-03-18T14:43:46.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.597+0000 7f709739d640 1 -- start start 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f709739d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70900b5cd0 0x7f7090160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f709739d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f709015dd50 0x7f709014cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f709739d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f709014d010 0x7f70900caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f709739d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70900d1fd0 con 0x7f709014d010 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f709739d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70900d1e50 con 0x7f70900b5cd0 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f709739d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f70900d2150 con 0x7f709015dd50 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f709014d010 0x7f70900caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f709639b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70900b5cd0 0x7f7090160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f709014d010 0x7f70900caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36064/0 (socket says 172.21.15.6:36064) 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 -- 172.21.15.6:0/1053593826 learned_addr learned my addr 172.21.15.6:0/1053593826 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7095b9a640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f709015dd50 0x7f709014cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 -- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f709015dd50 msgr2=0x7f709014cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f709015dd50 0x7f709014cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 -- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70900b5cd0 msgr2=0x7f7090160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70900b5cd0 0x7f7090160960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 -- 172.21.15.6:0/1053593826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709014d4c0 con 0x7f709014d010 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.598+0000 7f7096b9c640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f709014d010 0x7f70900caca0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f708802c4e0 tx=0x7f7088004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.599+0000 7f7086ffd640 1 -- 172.21.15.6:0/1053593826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088039050 con 0x7f709014d010 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.599+0000 7f7086ffd640 1 -- 172.21.15.6:0/1053593826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708803a040 con 0x7f709014d010 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.599+0000 7f7086ffd640 1 -- 172.21.15.6:0/1053593826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708803e610 con 0x7f709014d010 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.599+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70900cba20 con 0x7f709014d010 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.599+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70900cbfe0 con 0x7f709014d010 2024-03-18T14:43:46.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.600+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70900cdc40 con 0x7f709014d010 2024-03-18T14:43:46.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.600+0000 7f7086ffd640 1 -- 172.21.15.6:0/1053593826 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7088047020 con 0x7f709014d010 2024-03-18T14:43:46.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.600+0000 7f7086ffd640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707406bbd0 0x7f707406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.601+0000 7f7086ffd640 1 -- 172.21.15.6:0/1053593826 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f70880ace10 con 0x7f709014d010 2024-03-18T14:43:46.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.601+0000 7f709639b640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707406bbd0 0x7f707406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.601+0000 7f709639b640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707406bbd0 0x7f707406e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f708c006fd0 tx=0x7f708c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.605+0000 7f7086ffd640 1 -- 172.21.15.6:0/1053593826 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f708804e020 con 0x7f709014d010 2024-03-18T14:43:46.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.857+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f70900d1850 con 0x7f709014d010 2024-03-18T14:43:46.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.859+0000 7f7086ffd640 1 -- 172.21.15.6:0/1053593826 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f708807b120 con 0x7f709014d010 2024-03-18T14:43:46.861 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:43:46.862 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:43:46.862 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-18T14:43:46.862 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:43:46.862 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-18T14:43:46.862 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707406bbd0 msgr2=0x7f707406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707406bbd0 0x7f707406e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f708c006fd0 tx=0x7f708c008040 comp rx=0 tx=0).stop 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f709014d010 msgr2=0x7f70900caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f709014d010 0x7f70900caca0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f708802c4e0 tx=0x7f7088004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 shutdown_connections 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707406bbd0 0x7f707406e090 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f709014d010 0x7f70900caca0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f709015dd50 0x7f709014cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 --2- 172.21.15.6:0/1053593826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70900b5cd0 0x7f7090160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 >> 172.21.15.6:0/1053593826 conn(0x7f70900d05c0 msgr2=0x7f70900d3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:46.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 shutdown_connections 2024-03-18T14:43:46.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.861+0000 7f709739d640 1 -- 172.21.15.6:0/1053593826 wait complete. 2024-03-18T14:43:46.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:43:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.974+0000 7f81350ea640 1 Processor -- start 2024-03-18T14:43:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.974+0000 7f81350ea640 1 -- start start 2024-03-18T14:43:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f81350ea640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f813015e020 0x7f8130162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f81350ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81300b9660 con 0x7f81300ced00 2024-03-18T14:43:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f81350ea640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81300ce390 con 0x7f81300d3210 2024-03-18T14:43:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f81350ea640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f81300d2270 con 0x7f813015e020 2024-03-18T14:43:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f812ed9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f813015e020 0x7f8130162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f812e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81300ced00 0x7f81300d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58100/0 (socket says 172.21.15.6:58100) 2024-03-18T14:43:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f812ed9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f813015e020 0x7f8130162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36068/0 (socket says 172.21.15.6:36068) 2024-03-18T14:43:46.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.975+0000 7f812e59c640 1 -- 172.21.15.6:0/4014040840 learned_addr learned my addr 172.21.15.6:0/4014040840 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:46.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.976+0000 7f812ed9d640 1 -- 172.21.15.6:0/4014040840 >> v1:172.21.15.92:6789/0 conn(0x7f81300d3210 legacy=0x7f81300d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.976+0000 7f812ed9d640 1 -- 172.21.15.6:0/4014040840 >> v1:172.21.15.6:6789/0 conn(0x7f81300ced00 legacy=0x7f81300d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.976+0000 7f812ed9d640 1 -- 172.21.15.6:0/4014040840 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81300b9900 con 0x7f813015e020 2024-03-18T14:43:46.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.976+0000 7f812dd9b640 1 -- 172.21.15.6:0/4014040840 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493804435 0 0) 0x7f81300b9660 con 0x7f81300ced00 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.976+0000 7f812ed9d640 1 --2- 172.21.15.6:0/4014040840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813015e020 0x7f8130162380 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f81280097a0 tx=0x7f812802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c83b3e0234a193 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f812dd9b640 1 -- 172.21.15.6:0/4014040840 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128035020 con 0x7f813015e020 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f812dd9b640 1 -- 172.21.15.6:0/4014040840 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8128031830 con 0x7f813015e020 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f812dd9b640 1 -- 172.21.15.6:0/4014040840 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128031c00 con 0x7f813015e020 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f81350ea640 1 -- 172.21.15.6:0/4014040840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813015e020 msgr2=0x7f8130162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f81350ea640 1 --2- 172.21.15.6:0/4014040840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813015e020 0x7f8130162380 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f81280097a0 tx=0x7f812802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f81350ea640 1 -- 172.21.15.6:0/4014040840 shutdown_connections 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f81350ea640 1 --2- 172.21.15.6:0/4014040840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813015e020 0x7f8130162380 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f81350ea640 1 -- 172.21.15.6:0/4014040840 >> 172.21.15.6:0/4014040840 conn(0x7f8130072ce0 msgr2=0x7f81300730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:46.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f81350ea640 1 -- 172.21.15.6:0/4014040840 shutdown_connections 2024-03-18T14:43:46.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.977+0000 7f81350ea640 1 -- 172.21.15.6:0/4014040840 wait complete. 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 Processor -- start 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 -- start start 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81300ced00 0x7f8130152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81300d3210 0x7f813015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813015e020 0x7f813015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f81300cf2c0 con 0x7f81300d3210 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f81300ce890 con 0x7f813015e020 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f81350ea640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f81300cf440 con 0x7f81300ced00 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f812e59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81300d3210 0x7f813015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f812e59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81300d3210 0x7f813015c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36082/0 (socket says 172.21.15.6:36082) 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.979+0000 7f812e59c640 1 -- 172.21.15.6:0/407593536 learned_addr learned my addr 172.21.15.6:0/407593536 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812f59e640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813015e020 0x7f813015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812ed9d640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81300ced00 0x7f8130152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812e59c640 1 -- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81300ced00 msgr2=0x7f8130152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812e59c640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81300ced00 0x7f8130152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812e59c640 1 -- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813015e020 msgr2=0x7f813015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812e59c640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813015e020 0x7f813015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812e59c640 1 -- 172.21.15.6:0/407593536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813015e680 con 0x7f81300d3210 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812ed9d640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81300ced00 0x7f8130152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f812e59c640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81300d3210 0x7f813015c030 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f812400c460 tx=0x7f812400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f81177fe640 1 -- 172.21.15.6:0/407593536 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124005ce0 con 0x7f81300d3210 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f81177fe640 1 -- 172.21.15.6:0/407593536 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8124007070 con 0x7f81300d3210 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f813015e960 con 0x7f81300d3210 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f81177fe640 1 -- 172.21.15.6:0/407593536 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124007460 con 0x7f81300d3210 2024-03-18T14:43:46.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.980+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f813015ef20 con 0x7f81300d3210 2024-03-18T14:43:46.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.981+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f813015f120 con 0x7f81300d3210 2024-03-18T14:43:46.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.982+0000 7f81177fe640 1 -- 172.21.15.6:0/407593536 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8124007600 con 0x7f81300d3210 2024-03-18T14:43:46.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.982+0000 7f81177fe640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f811c06bbd0 0x7f811c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:46.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.983+0000 7f81177fe640 1 -- 172.21.15.6:0/407593536 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f812408a6f0 con 0x7f81300d3210 2024-03-18T14:43:46.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.983+0000 7f812ed9d640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f811c06bbd0 0x7f811c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:46.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.983+0000 7f812ed9d640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f811c06bbd0 0x7f811c06e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8128036000 tx=0x7f812803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:46.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:46.987+0000 7f81177fe640 1 -- 172.21.15.6:0/407593536 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f813015f120 con 0x7f81300d3210 2024-03-18T14:43:47.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.335+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f813008afe0 con 0x7f81300d3210 2024-03-18T14:43:47.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.336+0000 7f81177fe640 1 -- 172.21.15.6:0/407593536 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f81240590e0 con 0x7f81300d3210 2024-03-18T14:43:47.338 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:47.338 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/1499402578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1499402578},{"type":"v1","addr":"172.21.15.92:6836","nonce":1499402578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6022,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/300544617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":300544617},{"type":"v1","addr":"172.21.15.6:6835","nonce":300544617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/175151750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":175151750},{"type":"v1","addr":"172.21.15.92:6837","nonce":175151750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:43:41.338763+0000","modified":"2024-03-18T14:43:44.344988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6225},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6225":{"gid":6225,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/3070832840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3070832840},{"type":"v1","addr":"172.21.15.6:6834","nonce":3070832840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6225,"qdb_cluster":[6225]},"id":3}]} 2024-03-18T14:43:47.338 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-18T14:43:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f811c06bbd0 msgr2=0x7f811c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f811c06bbd0 0x7f811c06e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8128036000 tx=0x7f812803c040 comp rx=0 tx=0).stop 2024-03-18T14:43:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81300d3210 msgr2=0x7f813015c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81300d3210 0x7f813015c030 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f812400c460 tx=0x7f812400c920 comp rx=0 tx=0).stop 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 shutdown_connections 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f811c06bbd0 0x7f811c06e090 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813015e020 0x7f813015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81300d3210 0x7f813015c030 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 --2- 172.21.15.6:0/407593536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81300ced00 0x7f8130152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 >> 172.21.15.6:0/407593536 conn(0x7f8130072ce0 msgr2=0x7f81300cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 shutdown_connections 2024-03-18T14:43:47.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.338+0000 7f81350ea640 1 -- 172.21.15.6:0/407593536 wait complete. 2024-03-18T14:43:47.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T14:43:41.338763+0000', 'modified': '2024-03-18T14:43:44.344988+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6225}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6225': {'gid': 6225, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/3070832840', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 3070832840}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 3070832840}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6225, 'qdb_cluster': [6225]} 2024-03-18T14:43:47.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T14:43:47.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T14:43:47.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.855+0000 7f681410a640 1 Processor -- start 2024-03-18T14:43:47.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.855+0000 7f681410a640 1 -- start start 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f681410a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f681410a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f680c0b6df0 0x7f680c0b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f681410a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f680c0d3430 con 0x7f680c06c7a0 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f681410a640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f680c0d1f30 con 0x7f680c0b6df0 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f681410a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f680c0d2210 con 0x7f680c0d2fd0 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f68126a7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f680c0b6df0 0x7f680c0b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f6811ea6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f680c06c7a0 0x7f680c06cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58104/0 (socket says 172.21.15.6:58104) 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.856+0000 7f6811ea6640 1 -- 172.21.15.6:0/2814384817 learned_addr learned my addr 172.21.15.6:0/2814384817 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f68116a5640 1 --2- 172.21.15.6:0/2814384817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f68126a7640 1 -- 172.21.15.6:0/2814384817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 msgr2=0x7f680c0b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f68126a7640 1 --2- 172.21.15.6:0/2814384817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0b95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f68126a7640 1 -- 172.21.15.6:0/2814384817 >> v1:172.21.15.6:6789/0 conn(0x7f680c06c7a0 legacy=0x7f680c06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f68126a7640 1 -- 172.21.15.6:0/2814384817 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680c0d24f0 con 0x7f680c0b6df0 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f6810ea4640 1 -- 172.21.15.6:0/2814384817 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4266827254 0 0) 0x7f680c0d3430 con 0x7f680c06c7a0 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f68116a5640 1 --2- 172.21.15.6:0/2814384817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.857+0000 7f68126a7640 1 --2- 172.21.15.6:0/2814384817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c0b6df0 0x7f680c0b7250 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6804009d90 tx=0x7f680402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87695c260ec8bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f6810ea4640 1 -- 172.21.15.6:0/2814384817 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804039070 con 0x7f680c0b6df0 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f6810ea4640 1 -- 172.21.15.6:0/2814384817 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f680402cce0 con 0x7f680c0b6df0 2024-03-18T14:43:47.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f6810ea4640 1 -- 172.21.15.6:0/2814384817 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804034660 con 0x7f680c0b6df0 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 -- 172.21.15.6:0/2814384817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c0b6df0 msgr2=0x7f680c0b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 --2- 172.21.15.6:0/2814384817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c0b6df0 0x7f680c0b7250 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6804009d90 tx=0x7f680402bf30 comp rx=0 tx=0).stop 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 -- 172.21.15.6:0/2814384817 shutdown_connections 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 --2- 172.21.15.6:0/2814384817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c0b6df0 0x7f680c0b7250 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 --2- 172.21.15.6:0/2814384817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 -- 172.21.15.6:0/2814384817 >> 172.21.15.6:0/2814384817 conn(0x7f680c072ce0 msgr2=0x7f680c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 -- 172.21.15.6:0/2814384817 shutdown_connections 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.858+0000 7f681410a640 1 -- 172.21.15.6:0/2814384817 wait complete. 2024-03-18T14:43:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.859+0000 7f681410a640 1 Processor -- start 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.859+0000 7f681410a640 1 -- start start 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f681410a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c06c7a0 0x7f680c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f681410a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680c0b6df0 0x7f680c0b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f681410a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f681410a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f680c0b9b20 con 0x7f680c06c7a0 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f681410a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f680c0d2ab0 con 0x7f680c0b6df0 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f681410a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f680c0b9ca0 con 0x7f680c0d2fd0 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f6811ea6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c06c7a0 0x7f680c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f6811ea6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c06c7a0 0x7f680c0ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36106/0 (socket says 172.21.15.6:36106) 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f6811ea6640 1 -- 172.21.15.6:0/2941068441 learned_addr learned my addr 172.21.15.6:0/2941068441 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f68116a5640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680c0b6df0 0x7f680c0b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f68126a7640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.860+0000 7f6811ea6640 1 -- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 msgr2=0x7f680c0ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f6811ea6640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0ccf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f6811ea6640 1 -- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680c0b6df0 msgr2=0x7f680c0b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f6811ea6640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680c0b6df0 0x7f680c0b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f6811ea6640 1 -- 172.21.15.6:0/2941068441 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680c1f4870 con 0x7f680c06c7a0 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f68116a5640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680c0b6df0 0x7f680c0b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f6811ea6640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c06c7a0 0x7f680c0ca1f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f680800c430 tx=0x7f680800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f68027fc640 1 -- 172.21.15.6:0/2941068441 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68080074e0 con 0x7f680c06c7a0 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f68027fc640 1 -- 172.21.15.6:0/2941068441 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68080047e0 con 0x7f680c06c7a0 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680c1f3cf0 con 0x7f680c06c7a0 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f68027fc640 1 -- 172.21.15.6:0/2941068441 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6808004bd0 con 0x7f680c06c7a0 2024-03-18T14:43:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.861+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f680c1f42b0 con 0x7f680c06c7a0 2024-03-18T14:43:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.862+0000 7f68027fc640 1 -- 172.21.15.6:0/2941068441 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6808011070 con 0x7f680c06c7a0 2024-03-18T14:43:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.863+0000 7f68027fc640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67f006bbd0 0x7f67f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.863+0000 7f68027fc640 1 -- 172.21.15.6:0/2941068441 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f680808a7f0 con 0x7f680c06c7a0 2024-03-18T14:43:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.863+0000 7f68116a5640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67f006bbd0 0x7f67f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.864+0000 7f68116a5640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67f006bbd0 0x7f67f006e090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f67fc006fd0 tx=0x7f67fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.864+0000 7f6816dcd740 1 Processor -- start 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.864+0000 7f6816dcd740 1 -- start start 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980bfac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c2aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980bfb160 0x564980c2af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6816dcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564980b63c80 con 0x564980c28060 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564980b0caa0 con 0x564980c27c80 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x564980b5fd20 con 0x564980bfb160 2024-03-18T14:43:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c2aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c2aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36118/0 (socket says 172.21.15.6:36118) 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 -- 172.21.15.6:0/3235278822 learned_addr learned my addr 172.21.15.6:0/3235278822 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6811ea6640 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980bfac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68126a7640 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980bfb160 0x564980c2af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 -- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980bfb160 msgr2=0x564980c2af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980bfb160 0x564980c2af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 -- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 msgr2=0x564980bfac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980bfac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 -- 172.21.15.6:0/3235278822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564980b2eff0 con 0x564980c28060 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f6811ea6640 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980bfac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68126a7640 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980bfb160 0x564980c2af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f68116a5640 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c2aa10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00db60 tx=0x7f67fc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2f983056a58c4e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3235278822 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc019050 con 0x564980c28060 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3235278822 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67fc023070 con 0x564980c28060 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.865+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3235278822 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc012c90 con 0x564980c28060 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 -- 172.21.15.6:0/3235278822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 msgr2=0x564980c2aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c2aa10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00db60 tx=0x7f67fc010000 comp rx=0 tx=0).stop 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 -- 172.21.15.6:0/3235278822 shutdown_connections 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980bfb160 0x564980c2af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c2aa10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3235278822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980bfac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 -- 172.21.15.6:0/3235278822 >> 172.21.15.6:0/3235278822 conn(0x564980bfce90 msgr2=0x564980c45a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 -- 172.21.15.6:0/3235278822 shutdown_connections 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.866+0000 7f6816dcd740 1 -- 172.21.15.6:0/3235278822 wait complete. 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.867+0000 7f6816dcd740 1 -- start start 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.867+0000 7f6816dcd740 1 Processor -- start 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980c43fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c45840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c45d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564980bff530 con 0x564980c28060 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564980b1fef0 con 0x564980c25d50 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x564980c01d70 con 0x564980c27c80 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 10 client.0 _ll_register_callbacks cb 0x564980b1adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c45840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c45840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36120/0 (socket says 172.21.15.6:36120) 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 -- 172.21.15.6:0/904652383 learned_addr learned my addr 172.21.15.6:0/904652383 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6811ea6640 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980c43fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68126a7640 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c45d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 -- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 msgr2=0x564980c43fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980c43fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 -- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 msgr2=0x564980c45d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c45d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 -- 172.21.15.6:0/904652383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564980b2ca30 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f68116a5640 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c45840 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f67fc020040 tx=0x7f67fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b982321dbf4aa226 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f67ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f67ecff9640 1 -- 172.21.15.6:0/904652383 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc01e850 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f67ecff9640 1 -- 172.21.15.6:0/904652383 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f67fc033c30 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.868+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67fc023070 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f67ecff9640 1 -- 172.21.15.6:0/904652383 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f67fc019020 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 10 client.6378 fetch_fsmap learned FSMap version 25 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5649808782d0 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f67ecff9640 1 -- 172.21.15.6:0/904652383 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67fc023070 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f67ecff9640 1 -- 172.21.15.6:0/904652383 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f67fc01e9f0 con 0x564980c28060 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 10 client.6378 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 10 client.6378 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 20 client.6378 populate_metadata read hostname 'smithi006' 2024-03-18T14:43:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c55a90 0x564980c55e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 4 client.6378 mds_command: new command op to 6225 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 --> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564980b711c0 con 0x564980c55a90 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6811ea6640 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c55a90 0x564980c55e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f6811ea6640 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c55a90 0x564980c55e70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea0d5e487f21235 server_cookie=272e570a89f03406 in_seq=0 out_seq=0 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.869+0000 7f67ecff9640 10 client.6378 ms_handle_connect on v2:172.21.15.6:6832/3070832840 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f67ecff9640 1 -- 172.21.15.6:0/904652383 <== mds.0 v2:172.21.15.6:6832/3070832840 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564980b711c0 con 0x564980c55a90 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f67ecff9640 10 client.6378 handle_command_reply: tid=0 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 client.6378 shutdown 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 msgr2=0x564980c45840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c45840 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f67fc020040 tx=0x7f67fc004730 comp rx=0 tx=0).stop 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 shutdown_connections 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c55a90 0x564980c55e70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c45d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c45840 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f67ecff9640 0 client.6378 ms_handle_reset on v2:172.21.15.6:6832/3070832840 2024-03-18T14:43:47.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 --2- 172.21.15.6:0/904652383 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980c43fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 >> 172.21.15.6:0/904652383 conn(0x564980bfb160 msgr2=0x564980c48f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:47.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 shutdown_connections 2024-03-18T14:43:47.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.870+0000 7f6816dcd740 1 -- 172.21.15.6:0/904652383 wait complete. 2024-03-18T14:43:47.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.871+0000 7f6816dcd740 20 client.6378 trim_cache size 0 max 16384 2024-03-18T14:43:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.891+0000 7f6816dcd740 1 Processor -- start 2024-03-18T14:43:47.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.891+0000 7f6816dcd740 1 -- start start 2024-03-18T14:43:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c2aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980cb6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c48f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f6816dcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f680c1ff940 con 0x564980c28060 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f680c159060 con 0x564980c25d50 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f680c1ff2f0 con 0x564980c27c80 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f68126a7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c48f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f68126a7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c48f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36122/0 (socket says 172.21.15.6:36122) 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.892+0000 7f68126a7640 1 -- 172.21.15.6:0/1616507783 learned_addr learned my addr 172.21.15.6:0/1616507783 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f68116a5640 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980cb6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f6811ea6640 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c2aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f68126a7640 1 -- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 msgr2=0x564980cb6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f68126a7640 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980cb6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f68126a7640 1 -- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 msgr2=0x564980c2aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f68126a7640 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c2aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f68126a7640 1 -- 172.21.15.6:0/1616507783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649809426b0 con 0x564980c28060 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f68126a7640 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c48f20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f680402c540 tx=0x7f680402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e963e06292b50582 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f67eeffd640 1 -- 172.21.15.6:0/1616507783 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804039040 con 0x564980c28060 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f67eeffd640 1 -- 172.21.15.6:0/1616507783 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6804049680 con 0x564980c28060 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.893+0000 7f67eeffd640 1 -- 172.21.15.6:0/1616507783 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804049a70 con 0x564980c28060 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 -- 172.21.15.6:0/1616507783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 msgr2=0x564980c48f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c48f20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f680402c540 tx=0x7f680402e000 comp rx=0 tx=0).stop 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 -- 172.21.15.6:0/1616507783 shutdown_connections 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c28060 0x564980c48f20 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c27c80 0x564980cb6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 --2- 172.21.15.6:0/1616507783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c25d50 0x564980c2aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 -- 172.21.15.6:0/1616507783 >> 172.21.15.6:0/1616507783 conn(0x564980bfb160 msgr2=0x564980c27460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 -- 172.21.15.6:0/1616507783 shutdown_connections 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 -- 172.21.15.6:0/1616507783 wait complete. 2024-03-18T14:43:47.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 -- start start 2024-03-18T14:43:47.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.894+0000 7f6816dcd740 1 Processor -- start 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c25d50 0x564980c24df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980c25330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6816dcd740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c28060 0x564980cb3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6816dcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x564980c2e8c0 con 0x564980c25d50 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564980c2c2a0 con 0x564980c27c80 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6816dcd740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x564980c2cce0 con 0x564980c28060 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6816dcd740 10 client.0 _ll_register_callbacks cb 0x7f67fc00da30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T14:43:47.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6811ea6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c25d50 0x564980c24df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6811ea6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c25d50 0x564980c24df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36126/0 (socket says 172.21.15.6:36126) 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f6811ea6640 1 -- 172.21.15.6:0/3491849969 learned_addr learned my addr 172.21.15.6:0/3491849969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f68126a7640 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c28060 0x564980cb3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.895+0000 7f68116a5640 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980c25330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6811ea6640 1 -- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c28060 msgr2=0x564980cb3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6811ea6640 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c28060 0x564980cb3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6811ea6640 1 -- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 msgr2=0x564980c25330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6811ea6640 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980c25330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6811ea6640 1 -- 172.21.15.6:0/3491849969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564980c42be0 con 0x564980c25d50 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6811ea6640 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c25d50 0x564980c24df0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6808058a70 tx=0x7f680808a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e082c953d5b593 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f67ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f67ecff9640 1 -- 172.21.15.6:0/3491849969 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6808004d70 con 0x564980c25d50 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f67ecff9640 1 -- 172.21.15.6:0/3491849969 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f6808083dc0 con 0x564980c25d50 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6804049680 con 0x564980c25d50 2024-03-18T14:43:47.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f67ecff9640 1 -- 172.21.15.6:0/3491849969 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f680801ad60 con 0x564980c25d50 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 10 client.6384 fetch_fsmap learned FSMap version 25 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x564980cb5610 con 0x564980c25d50 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f67ecff9640 1 -- 172.21.15.6:0/3491849969 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6804049680 con 0x564980c25d50 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f67ecff9640 1 -- 172.21.15.6:0/3491849969 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f6808055150 con 0x564980c25d50 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 10 client.6384 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 10 client.6384 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 20 client.6384 populate_metadata read hostname 'smithi006' 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c680f0 0x564980c684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 4 client.6384 mds_command: new command op to 6225 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.896+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 --> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] -- command(tid 0: {"prefix": "status"}) -- 0x564980b6b510 con 0x564980c680f0 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f68116a5640 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c680f0 0x564980c684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f67ecff9640 10 client.6384 ms_handle_connect on v2:172.21.15.6:6832/3070832840 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f68116a5640 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c680f0 0x564980c684d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19d288b4b2d07f76 server_cookie=bf96fd2ba535846f in_seq=0 out_seq=0 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f67ecff9640 1 -- 172.21.15.6:0/3491849969 <== mds.0 v2:172.21.15.6:6832/3070832840 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x564980b6b510 con 0x564980c680f0 2024-03-18T14:43:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f67ecff9640 10 client.6384 handle_command_reply: tid=0 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 client.6384 shutdown 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c25d50 msgr2=0x564980c24df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c25d50 0x564980c24df0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6808058a70 tx=0x7f680808a040 comp rx=0 tx=0).stop 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 shutdown_connections 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:6832/3070832840,v1:172.21.15.6:6834/3070832840] conn(0x564980c680f0 0x564980c684d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x564980c28060 0x564980cb3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564980c27c80 0x564980c25330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f67ecff9640 0 client.6384 ms_handle_reset on v2:172.21.15.6:6832/3070832840 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 --2- 172.21.15.6:0/3491849969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564980c25d50 0x564980c24df0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 >> 172.21.15.6:0/3491849969 conn(0x564980bfb160 msgr2=0x564980c56150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 shutdown_connections 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.897+0000 7f6816dcd740 1 -- 172.21.15.6:0/3491849969 wait complete. 2024-03-18T14:43:47.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f6816dcd740 20 client.6384 trim_cache size 0 max 16384 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "a75a8f99-9e80-46f7-9da4-368af92e8063", 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 6225, 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 5.9186649559999998, 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 25, 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 56, 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 56, 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 10.946643116000001 2024-03-18T14:43:47.906 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67f006bbd0 msgr2=0x7f67f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67f006bbd0 0x7f67f006e090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f67fc006fd0 tx=0x7f67fc008040 comp rx=0 tx=0).stop 2024-03-18T14:43:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c06c7a0 msgr2=0x7f680c0ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c06c7a0 0x7f680c0ca1f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f680800c430 tx=0x7f680800c8f0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 shutdown_connections 2024-03-18T14:43:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f67f006bbd0 0x7f67f006e090 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680c0d2fd0 0x7f680c0ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680c0b6df0 0x7f680c0b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 --2- 172.21.15.6:0/2941068441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f680c06c7a0 0x7f680c0ca1f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 >> 172.21.15.6:0/2941068441 conn(0x7f680c072ce0 msgr2=0x7f680c0b71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 shutdown_connections 2024-03-18T14:43:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:47.898+0000 7f681410a640 1 -- 172.21.15.6:0/2941068441 wait complete. 2024-03-18T14:43:47.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:43:47.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-18T14:43:47.949+0000 7f9fe9ee7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T14:43:48.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.010+0000 7fcd5ae0c640 1 Processor -- start 2024-03-18T14:43:48.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.010+0000 7fcd5ae0c640 1 -- start start 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.011+0000 7fcd5ae0c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcd5415e240 0x7fcd540cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.011+0000 7fcd5ae0c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd540d1920 con 0x7fcd5415e620 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd5ae0c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd540cb760 con 0x7fcd540cf4f0 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd5ae0c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcd540cba40 con 0x7fcd5415e240 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd53fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcd5415e620 0x7fcd540cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58108/0 (socket says 172.21.15.6:58108) 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd53fff640 1 -- 172.21.15.6:0/399457203 learned_addr learned my addr 172.21.15.6:0/399457203 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd58ba8640 1 --2- 172.21.15.6:0/399457203 >> v2:172.21.15.92:3300/0 conn(0x7fcd5415e240 0x7fcd540cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761643288 0 0) 0x7fcd540d1920 con 0x7fcd5415e620 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd40003520 con 0x7fcd5415e620 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4038216466 0 0) 0x7fcd40003520 con 0x7fcd5415e620 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 >> v2:172.21.15.92:3300/0 conn(0x7fcd5415e240 msgr2=0x7fcd540cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 --2- 172.21.15.6:0/399457203 >> v2:172.21.15.92:3300/0 conn(0x7fcd5415e240 0x7fcd540cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 >> v1:172.21.15.92:6789/0 conn(0x7fcd540cf4f0 legacy=0x7fcd540b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd540cbd20 con 0x7fcd5415e620 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fcd440019f0 con 0x7fcd5415e620 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 >> v1:172.21.15.6:6789/0 conn(0x7fcd5415e620 legacy=0x7fcd540cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd400039b0 0x7fcd40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd40004800 0x7fcd40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd400051c0 0x7fcd4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540cba40 con 0x7fcd40004800 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4000c060 con 0x7fcd400051c0 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcd4000c260 con 0x7fcd400039b0 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.012+0000 7fcd53fff640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd40004800 0x7fcd40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd58ba8640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd400039b0 0x7fcd40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd593a9640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd400051c0 0x7fcd4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd53fff640 1 -- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd400039b0 msgr2=0x7fcd40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:48.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd53fff640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd400039b0 0x7fcd40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd53fff640 1 -- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd400051c0 msgr2=0x7fcd4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd53fff640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd400051c0 0x7fcd4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd593a9640 1 -- 172.21.15.6:0/399457203 reap_dead start 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd53fff640 1 -- 172.21.15.6:0/399457203 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd40003520 con 0x7fcd40004800 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.013+0000 7fcd53fff640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd40004800 0x7fcd40004c80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fcd5415ea90 tx=0x7fcd40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b05f00599b8141 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4400a4a0 con 0x7fcd40004800 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd44034650 con 0x7fcd40004800 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd537fe640 1 -- 172.21.15.6:0/399457203 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd44034a80 con 0x7fcd40004800 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/399457203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd40004800 msgr2=0x7fcd40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd40004800 0x7fcd40004c80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fcd5415ea90 tx=0x7fcd40005830 comp rx=0 tx=0).stop 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/399457203 shutdown_connections 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/399457203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd40004800 0x7fcd40004c80 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/399457203 >> 172.21.15.6:0/399457203 conn(0x7fcd54099a50 msgr2=0x7fcd540d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:48.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/399457203 shutdown_connections 2024-03-18T14:43:48.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.014+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/399457203 wait complete. 2024-03-18T14:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.015+0000 7fcd5ae0c640 1 Processor -- start 2024-03-18T14:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.015+0000 7fcd5ae0c640 1 -- start start 2024-03-18T14:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd5ae0c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd40004800 0x7fcd540b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd5ae0c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd541ef2e0 0x7fcd5409e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd5ae0c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd541f2300 0x7fcd541f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd5ae0c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540cdc50 con 0x7fcd541f2300 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd5ae0c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd540cdad0 con 0x7fcd541ef2e0 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd5ae0c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcd540cddd0 con 0x7fcd40004800 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd593a9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd541f2300 0x7fcd541f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd593a9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd541f2300 0x7fcd541f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36142/0 (socket says 172.21.15.6:36142) 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.016+0000 7fcd593a9640 1 -- 172.21.15.6:0/3909142811 learned_addr learned my addr 172.21.15.6:0/3909142811 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd58ba8640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd40004800 0x7fcd540b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd53fff640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd541ef2e0 0x7fcd5409e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd593a9640 1 -- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd40004800 msgr2=0x7fcd540b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd593a9640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd40004800 0x7fcd540b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd593a9640 1 -- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd541ef2e0 msgr2=0x7fcd5409e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd593a9640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd541ef2e0 0x7fcd5409e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd593a9640 1 -- 172.21.15.6:0/3909142811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd540b5740 con 0x7fcd541f2300 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd593a9640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd541f2300 0x7fcd541f2760 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fcd540b0340 tx=0x7fcd4c00f7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:48.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd517fa640 1 -- 172.21.15.6:0/3909142811 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c01a070 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd517fa640 1 -- 172.21.15.6:0/3909142811 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd4c005320 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd540b5a20 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd517fa640 1 -- 172.21.15.6:0/3909142811 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c007270 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.017+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd541f1de0 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.018+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd541f1fe0 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.019+0000 7fcd517fa640 1 -- 172.21.15.6:0/3909142811 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcd4c01d070 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.019+0000 7fcd517fa640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcd3406bbd0 0x7fcd3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.019+0000 7fcd517fa640 1 -- 172.21.15.6:0/3909142811 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fcd4c089bc0 con 0x7fcd541f2300 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.020+0000 7fcd58ba8640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcd3406bbd0 0x7fcd3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.020+0000 7fcd58ba8640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcd3406bbd0 0x7fcd3406e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcd540ce590 tx=0x7fcd3c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:48.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.024+0000 7fcd517fa640 1 -- 172.21.15.6:0/3909142811 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcd541f1fe0 con 0x7fcd541f2300 2024-03-18T14:43:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.245+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcd541f1bf0 con 0x7fcd541f2300 2024-03-18T14:43:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.246+0000 7fcd517fa640 1 -- 172.21.15.6:0/3909142811 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7fcd4c057f60 con 0x7fcd541f2300 2024-03-18T14:43:48.248 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T14:43:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcd3406bbd0 msgr2=0x7fcd3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcd3406bbd0 0x7fcd3406e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcd540ce590 tx=0x7fcd3c009190 comp rx=0 tx=0).stop 2024-03-18T14:43:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd541f2300 msgr2=0x7fcd541f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd541f2300 0x7fcd541f2760 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fcd540b0340 tx=0x7fcd4c00f7d0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 shutdown_connections 2024-03-18T14:43:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcd3406bbd0 0x7fcd3406e090 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd541f2300 0x7fcd541f2760 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd541ef2e0 0x7fcd5409e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 --2- 172.21.15.6:0/3909142811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd40004800 0x7fcd540b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 >> 172.21.15.6:0/3909142811 conn(0x7fcd54099a50 msgr2=0x7fcd540b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 shutdown_connections 2024-03-18T14:43:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48.248+0000 7fcd5ae0c640 1 -- 172.21.15.6:0/3909142811 wait complete. 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:43:48.261 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:48.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:48.262 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:48.316 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42537sec preferred_lft 42537sec 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:48.317 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c41d:adff:fe29:e32/64 scope link 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::d035:11ff:fe58:6e4e/64 scope link 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::9023:d7ff:fecf:2a25/64 scope link 2024-03-18T14:43:48.318 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:48.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:48.319 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:43:48.381 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:43:48.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:48.383 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:43:48.438 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:43:48.438 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:43:48.438 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:43:48.439 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:43:48.439 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:43:48.439 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:43:48.439 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:43:48.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:43:48.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:43:48.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:43:48.439 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:43:48.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:43:48.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:48.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:48.574 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:43:48.589 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T14:43:48.589 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:43:48.589 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:48.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:48.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:43:48.655 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T14:43:48.656 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:48.656 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:48.710 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:48.710 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:48.710 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:48.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:48.835 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:48.890 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:48.890 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:48.890 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:48.890 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:48.890 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:48.890 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:48.891 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:48.892 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:48.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:48.893 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:48.944 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:43:48.944 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T14:43:48.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:49.013 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:49.070 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:49.071 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:49.072 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:49.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:49.073 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:49.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:2024-03-18T14:43:49.071+0000 7f6f12c4c480 -1 init, newargv = 0x56350a873180 newargc=15 2024-03-18T14:43:49.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[59919]: starting ceph client 2024-03-18T14:43:49.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[59919]: starting fuse 2024-03-18T14:43:49.093 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:43:49.093 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T14:43:49.093 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:43:49.094 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:43:49.095 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:49.095 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.0")) 2024-03-18T14:43:49.095 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:43:49.213 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.0.59919.asok 2024-03-18T14:43:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:49.224 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59919.asok 2024-03-18T14:43:49.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:49.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59919.asok status 2024-03-18T14:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "0", 2024-03-18T14:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:43:49.362 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "59919", 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 6396, 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6396 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:49.363 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 679561079 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 679561079 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.6396 192.168.0.1:0/679561079", 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/679561079", 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 25, 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 56, 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:43:49.364 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:43:49.365 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:43:49.365 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:49.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:49.370 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:43:49.370 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:49.386 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:43:49.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:49.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:49.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:49.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:49.453 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2024-03-18T14:43:49.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2024-03-18T14:43:49.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.658+0000 7f88655df640 1 Processor -- start 2024-03-18T14:43:49.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f88655df640 1 -- start start 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f88655df640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88600ce9d0 0x7f88600cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f88655df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d2510 0x7f88600d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f88655df640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88600d2970 con 0x7f88600d0560 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f88655df640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f88600d09d0 con 0x7f88600ce9d0 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f88655df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f88600cd8d0 con 0x7f88600d2510 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f885effd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88600ce9d0 0x7f88600cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.659+0000 7f885effd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88600ce9d0 0x7f88600cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36144/0 (socket says 172.21.15.6:36144) 2024-03-18T14:43:49.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.660+0000 7f885effd640 1 -- 172.21.15.6:0/2175050108 learned_addr learned my addr 172.21.15.6:0/2175050108 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.660+0000 7f8857fff640 1 --2- 172.21.15.6:0/2175050108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d2510 0x7f88600d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.660+0000 7f885effd640 1 -- 172.21.15.6:0/2175050108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d2510 msgr2=0x7f88600d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.660+0000 7f885effd640 1 --2- 172.21.15.6:0/2175050108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d2510 0x7f88600d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.660+0000 7f885effd640 1 -- 172.21.15.6:0/2175050108 >> v1:172.21.15.92:6789/0 conn(0x7f88600d0560 legacy=0x7f886015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.660+0000 7f885effd640 1 -- 172.21.15.6:0/2175050108 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88600cfac0 con 0x7f88600ce9d0 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f885effd640 1 --2- 172.21.15.6:0/2175050108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f88600cc650 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8850009850 tx=0x7f885002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82f1175b38ce62b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f885e7fc640 1 -- 172.21.15.6:0/2175050108 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850004230 con 0x7f88600ce9d0 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f885e7fc640 1 -- 172.21.15.6:0/2175050108 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88500043d0 con 0x7f88600ce9d0 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f885e7fc640 1 -- 172.21.15.6:0/2175050108 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850031830 con 0x7f88600ce9d0 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 -- 172.21.15.6:0/2175050108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 msgr2=0x7f88600cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 --2- 172.21.15.6:0/2175050108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f88600cc650 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8850009850 tx=0x7f885002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 -- 172.21.15.6:0/2175050108 shutdown_connections 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 --2- 172.21.15.6:0/2175050108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d2510 0x7f88600d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 --2- 172.21.15.6:0/2175050108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f88600cc650 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 -- 172.21.15.6:0/2175050108 >> 172.21.15.6:0/2175050108 conn(0x7f886008b0c0 msgr2=0x7f88600d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 -- 172.21.15.6:0/2175050108 shutdown_connections 2024-03-18T14:43:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.661+0000 7f88655df640 1 -- 172.21.15.6:0/2175050108 wait complete. 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.662+0000 7f88655df640 1 Processor -- start 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.662+0000 7f88655df640 1 -- start start 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f88655df640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f886014c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f88655df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d0560 0x7f886006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f88655df640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88600d2510 0x7f886014bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f88655df640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f88600cd280 con 0x7f88600ce9d0 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f88655df640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88600ccfa0 con 0x7f88600d2510 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f88655df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f88600cd560 con 0x7f88600d0560 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f885effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f886014c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f885effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f886014c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36148/0 (socket says 172.21.15.6:36148) 2024-03-18T14:43:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f885effd640 1 -- 172.21.15.6:0/602780936 learned_addr learned my addr 172.21.15.6:0/602780936 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f8857fff640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d0560 0x7f886006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.663+0000 7f885f7fe640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88600d2510 0x7f886014bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f885effd640 1 -- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d0560 msgr2=0x7f886006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f885effd640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d0560 0x7f886006c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f885effd640 1 -- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88600d2510 msgr2=0x7f886014bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f885effd640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88600d2510 0x7f886014bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f885effd640 1 -- 172.21.15.6:0/602780936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88600b6700 con 0x7f88600ce9d0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f885f7fe640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88600d2510 0x7f886014bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f885effd640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f886014c960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8850009970 tx=0x7f88500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f88577fe640 1 -- 172.21.15.6:0/602780936 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885003f070 con 0x7f88600ce9d0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f88577fe640 1 -- 172.21.15.6:0/602780936 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8850031a40 con 0x7f88600ce9d0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88600b6420 con 0x7f88600ce9d0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f88577fe640 1 -- 172.21.15.6:0/602780936 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850031e10 con 0x7f88600ce9d0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.664+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88600ca630 con 0x7f88600ce9d0 2024-03-18T14:43:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.665+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88600751b0 con 0x7f88600ce9d0 2024-03-18T14:43:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.666+0000 7f88577fe640 1 -- 172.21.15.6:0/602780936 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8850003610 con 0x7f88600ce9d0 2024-03-18T14:43:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.666+0000 7f88577fe640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f884006bbd0 0x7f884006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.666+0000 7f88577fe640 1 -- 172.21.15.6:0/602780936 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f88500ace30 con 0x7f88600ce9d0 2024-03-18T14:43:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.666+0000 7f8857fff640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f884006bbd0 0x7f884006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.667+0000 7f8857fff640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f884006bbd0 0x7f884006e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f88600b2350 tx=0x7f8848008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:49.670+0000 7f88577fe640 1 -- 172.21.15.6:0/602780936 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f885007b790 con 0x7f88600ce9d0 2024-03-18T14:43:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.052+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) -- 0x7f88600c5310 con 0x7f88600ce9d0 2024-03-18T14:43:50.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.054+0000 7f88577fe640 1 -- 172.21.15.6:0/602780936 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v16) ==== 183+0+140 (secure 0 0 0) 0x7f885007b090 con 0x7f88600ce9d0 2024-03-18T14:43:50.057 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:43:50.057 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:43:50.057 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-18T14:43:50.057 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:43:50.057 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow r pool=cephfs_data" 2024-03-18T14:43:50.057 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f884006bbd0 msgr2=0x7f884006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f884006bbd0 0x7f884006e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f88600b2350 tx=0x7f8848008040 comp rx=0 tx=0).stop 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 msgr2=0x7f886014c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f886014c960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8850009970 tx=0x7f88500046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 shutdown_connections 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f884006bbd0 0x7f884006e090 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88600d2510 0x7f886014bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88600d0560 0x7f886006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 --2- 172.21.15.6:0/602780936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88600ce9d0 0x7f886014c960 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 >> 172.21.15.6:0/602780936 conn(0x7f886008b0c0 msgr2=0x7f88600b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 shutdown_connections 2024-03-18T14:43:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.057+0000 7f88655df640 1 -- 172.21.15.6:0/602780936 wait complete. 2024-03-18T14:43:50.070 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:50.070 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:50.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.247+0000 7fb1b6844640 1 Processor -- start 2024-03-18T14:43:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.247+0000 7fb1b6844640 1 -- start start 2024-03-18T14:43:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1b6844640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1b015e170 0x7fb1b00b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1b6844640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b00cef50 con 0x7fb1b00d0d90 2024-03-18T14:43:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1b6844640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b00cf230 con 0x7fb1b00d3330 2024-03-18T14:43:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1b6844640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb1b00b9b90 con 0x7fb1b015e170 2024-03-18T14:43:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1affff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1b015e170 0x7fb1b00b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1b4de1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb1b00d0d90 0x7fb1b00b6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58116/0 (socket says 172.21.15.6:58116) 2024-03-18T14:43:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1b4de1640 1 -- 172.21.15.6:0/2001987304 learned_addr learned my addr 172.21.15.6:0/2001987304 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:50.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1affff640 1 -- 172.21.15.6:0/2001987304 >> v1:172.21.15.92:6789/0 conn(0x7fb1b00d3330 legacy=0x7fb1b00d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:50.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1affff640 1 -- 172.21.15.6:0/2001987304 >> v1:172.21.15.6:6789/0 conn(0x7fb1b00d0d90 legacy=0x7fb1b00b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1aeffd640 1 -- 172.21.15.6:0/2001987304 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377557750 0 0) 0x7fb1b00cef50 con 0x7fb1b00d0d90 2024-03-18T14:43:50.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.248+0000 7fb1affff640 1 -- 172.21.15.6:0/2001987304 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b00d2b30 con 0x7fb1b015e170 2024-03-18T14:43:50.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.249+0000 7fb1affff640 1 --2- 172.21.15.6:0/2001987304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b015e170 0x7fb1b00b9650 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00097a0 tx=0x7fb1a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88514ba90139b139 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.249+0000 7fb1aeffd640 1 -- 172.21.15.6:0/2001987304 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0035020 con 0x7fb1b015e170 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.249+0000 7fb1aeffd640 1 -- 172.21.15.6:0/2001987304 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a0031830 con 0x7fb1b015e170 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.249+0000 7fb1aeffd640 1 -- 172.21.15.6:0/2001987304 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0031c00 con 0x7fb1b015e170 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.250+0000 7fb1b6844640 1 -- 172.21.15.6:0/2001987304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b015e170 msgr2=0x7fb1b00b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.250+0000 7fb1b6844640 1 --2- 172.21.15.6:0/2001987304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b015e170 0x7fb1b00b9650 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00097a0 tx=0x7fb1a002b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.250+0000 7fb1b6844640 1 -- 172.21.15.6:0/2001987304 shutdown_connections 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.250+0000 7fb1b6844640 1 --2- 172.21.15.6:0/2001987304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b015e170 0x7fb1b00b9650 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.250+0000 7fb1b6844640 1 -- 172.21.15.6:0/2001987304 >> 172.21.15.6:0/2001987304 conn(0x7fb1b006c900 msgr2=0x7fb1b006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.250+0000 7fb1b6844640 1 -- 172.21.15.6:0/2001987304 shutdown_connections 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.250+0000 7fb1b6844640 1 -- 172.21.15.6:0/2001987304 wait complete. 2024-03-18T14:43:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.251+0000 7fb1b6844640 1 Processor -- start 2024-03-18T14:43:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.251+0000 7fb1b6844640 1 -- start start 2024-03-18T14:43:50.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1b6844640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00d0d90 0x7fb1b0158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1b6844640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1b00d3330 0x7fb1b00cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1b6844640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1b015e170 0x7fb1b01f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1b6844640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b00ceb70 con 0x7fb1b00d0d90 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1b6844640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b00d2820 con 0x7fb1b015e170 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1b6844640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b00cecf0 con 0x7fb1b00d3330 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00d0d90 0x7fb1b0158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00d0d90 0x7fb1b0158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36168/0 (socket says 172.21.15.6:36168) 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 -- 172.21.15.6:0/3343515932 learned_addr learned my addr 172.21.15.6:0/3343515932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1b4de1640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1b015e170 0x7fb1b01f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1af7fe640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1b00d3330 0x7fb1b00cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 -- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1b00d3330 msgr2=0x7fb1b00cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1b00d3330 0x7fb1b00cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 -- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1b015e170 msgr2=0x7fb1b01f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1b015e170 0x7fb1b01f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.252+0000 7fb1affff640 1 -- 172.21.15.6:0/3343515932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b00ccd50 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.253+0000 7fb1b4de1640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1b015e170 0x7fb1b01f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.253+0000 7fb1affff640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00d0d90 0x7fb1b0158280 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00097a0 tx=0x7fb1a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.253+0000 7fb1acff9640 1 -- 172.21.15.6:0/3343515932 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0004230 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.253+0000 7fb1acff9640 1 -- 172.21.15.6:0/3343515932 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a00043d0 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.253+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b01f4500 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.253+0000 7fb1acff9640 1 -- 172.21.15.6:0/3343515932 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a003cad0 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.253+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b00cc010 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.254+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b008afe0 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.254+0000 7fb1acff9640 1 -- 172.21.15.6:0/3343515932 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb1a0038020 con 0x7fb1b00d0d90 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.255+0000 7fb1acff9640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb19c06bbd0 0x7fb19c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.255+0000 7fb1acff9640 1 -- 172.21.15.6:0/3343515932 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fb1a00ace10 con 0x7fb1b00d0d90 2024-03-18T14:43:50.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.255+0000 7fb1af7fe640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb19c06bbd0 0x7fb19c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.255+0000 7fb1af7fe640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb19c06bbd0 0x7fb19c06e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00d1fb0 tx=0x7fb1a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.259+0000 7fb1acff9640 1 -- 172.21.15.6:0/3343515932 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb1a0041070 con 0x7fb1b00d0d90 2024-03-18T14:43:50.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.441+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1b004f620 con 0x7fb1b00d0d90 2024-03-18T14:43:50.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.443+0000 7fb1acff9640 1 -- 172.21.15.6:0/3343515932 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7fb1a0045e50 con 0x7fb1b00d0d90 2024-03-18T14:43:50.445 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:50.446 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:50.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.444+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb19c06bbd0 msgr2=0x7fb19c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb19c06bbd0 0x7fb19c06e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00d1fb0 tx=0x7fb1a4009040 comp rx=0 tx=0).stop 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00d0d90 msgr2=0x7fb1b0158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00d0d90 0x7fb1b0158280 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00097a0 tx=0x7fb1a00046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 shutdown_connections 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb19c06bbd0 0x7fb19c06e090 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1b015e170 0x7fb1b01f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1b00d3330 0x7fb1b00cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 --2- 172.21.15.6:0/3343515932 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00d0d90 0x7fb1b0158280 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 >> 172.21.15.6:0/3343515932 conn(0x7fb1b006c900 msgr2=0x7fb1b00b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:50.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 shutdown_connections 2024-03-18T14:43:50.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.445+0000 7fb1b6844640 1 -- 172.21.15.6:0/3343515932 wait complete. 2024-03-18T14:43:50.458 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:43:39.597879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:43:40.299491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:50.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.561+0000 7fe9f9a27640 1 Processor -- start 2024-03-18T14:43:50.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.561+0000 7fe9f9a27640 1 -- start start 2024-03-18T14:43:50.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.562+0000 7fe9f9a27640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40ce970 0x7fe9f40ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.562+0000 7fe9f9a27640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe9f40d2630 0x7fe9f40d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.562+0000 7fe9f9a27640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9f40cf290 con 0x7fe9f40b9940 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.562+0000 7fe9f9a27640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe9f40d21a0 con 0x7fe9f40d2630 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.562+0000 7fe9f9a27640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f415eb20 con 0x7fe9f40ce970 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f2ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40ce970 0x7fe9f40ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f37fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe9f40d2630 0x7fe9f40d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f2ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40ce970 0x7fe9f40ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55410/0 (socket says 172.21.15.6:55410) 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f2ffd640 1 -- 172.21.15.6:0/3021862304 learned_addr learned my addr 172.21.15.6:0/3021862304 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f37fe640 1 -- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40ce970 msgr2=0x7fe9f40ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f37fe640 1 --2- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40ce970 0x7fe9f40ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f37fe640 1 -- 172.21.15.6:0/3021862304 >> v1:172.21.15.92:6789/0 conn(0x7fe9f40b9940 legacy=0x7fe9f415e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f37fe640 1 -- 172.21.15.6:0/3021862304 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f415edd0 con 0x7fe9f40d2630 2024-03-18T14:43:50.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.563+0000 7fe9f2ffd640 1 --2- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40ce970 0x7fe9f40ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.564+0000 7fe9f37fe640 1 --2- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40d2630 0x7fe9f40d2ac0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009d90 tx=0x7fe9ec02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3e7398db34cc0b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.564+0000 7fe9f27fc640 1 -- 172.21.15.6:0/3021862304 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec039070 con 0x7fe9f40d2630 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.564+0000 7fe9f27fc640 1 -- 172.21.15.6:0/3021862304 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec02cc50 con 0x7fe9f40d2630 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.564+0000 7fe9f27fc640 1 -- 172.21.15.6:0/3021862304 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec034440 con 0x7fe9f40d2630 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40d2630 msgr2=0x7fe9f40d2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40d2630 0x7fe9f40d2ac0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009d90 tx=0x7fe9ec02bea0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3021862304 shutdown_connections 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40d2630 0x7fe9f40d2ac0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3021862304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40ce970 0x7fe9f40ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3021862304 >> 172.21.15.6:0/3021862304 conn(0x7fe9f406c900 msgr2=0x7fe9f406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:50.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3021862304 shutdown_connections 2024-03-18T14:43:50.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.565+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3021862304 wait complete. 2024-03-18T14:43:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.566+0000 7fe9f9a27640 1 Processor -- start 2024-03-18T14:43:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.566+0000 7fe9f9a27640 1 -- start start 2024-03-18T14:43:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40b9940 0x7fe9f41583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40ce970 0x7fe9f4159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9f40d2630 0x7fe9f415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f415f390 con 0x7fe9f40d2630 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f40d1e30 con 0x7fe9f40ce970 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f415bd70 con 0x7fe9f40b9940 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9f40d2630 0x7fe9f415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9f40d2630 0x7fe9f415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36170/0 (socket says 172.21.15.6:36170) 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 -- 172.21.15.6:0/3765097818 learned_addr learned my addr 172.21.15.6:0/3765097818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9ebfff640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40ce970 0x7fe9f4159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f2ffd640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40b9940 0x7fe9f41583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 -- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40b9940 msgr2=0x7fe9f41583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40b9940 0x7fe9f41583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 -- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40ce970 msgr2=0x7fe9f4159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40ce970 0x7fe9f4159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 -- 172.21.15.6:0/3765097818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f415aa90 con 0x7fe9f40d2630 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f37fe640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9f40d2630 0x7fe9f415a380 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009d90 tx=0x7fe9ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f0ff9640 1 -- 172.21.15.6:0/3765097818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec039050 con 0x7fe9f40d2630 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f0ff9640 1 -- 172.21.15.6:0/3765097818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec045070 con 0x7fe9f40d2630 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f40cb550 con 0x7fe9f40d2630 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f0ff9640 1 -- 172.21.15.6:0/3765097818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec009450 con 0x7fe9f40d2630 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.567+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f4158920 con 0x7fe9f40d2630 2024-03-18T14:43:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.568+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f40751b0 con 0x7fe9f40d2630 2024-03-18T14:43:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.569+0000 7fe9f0ff9640 1 -- 172.21.15.6:0/3765097818 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe9ec03e610 con 0x7fe9f40d2630 2024-03-18T14:43:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.569+0000 7fe9f0ff9640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe9d406bbd0 0x7fe9d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.570+0000 7fe9f0ff9640 1 -- 172.21.15.6:0/3765097818 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fe9ec0ad4c0 con 0x7fe9f40d2630 2024-03-18T14:43:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.570+0000 7fe9f2ffd640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe9d406bbd0 0x7fe9d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.570+0000 7fe9f2ffd640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe9d406bbd0 0x7fe9d406e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4006ca0 tx=0x7fe9e4009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.574+0000 7fe9f0ff9640 1 -- 172.21.15.6:0/3765097818 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe9ec07b860 con 0x7fe9f40d2630 2024-03-18T14:43:50.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.760+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe9f415e800 con 0x7fe9f40d2630 2024-03-18T14:43:50.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.760+0000 7fe9f0ff9640 1 -- 172.21.15.6:0/3765097818 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fe9ec07f900 con 0x7fe9f40d2630 2024-03-18T14:43:50.763 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe9d406bbd0 msgr2=0x7fe9d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe9d406bbd0 0x7fe9d406e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4006ca0 tx=0x7fe9e4009240 comp rx=0 tx=0).stop 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9f40d2630 msgr2=0x7fe9f415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9f40d2630 0x7fe9f415a380 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009d90 tx=0x7fe9ec004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 shutdown_connections 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe9d406bbd0 0x7fe9d406e090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9f40d2630 0x7fe9f415a380 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9f40ce970 0x7fe9f4159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 --2- 172.21.15.6:0/3765097818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9f40b9940 0x7fe9f41583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 >> 172.21.15.6:0/3765097818 conn(0x7fe9f406c900 msgr2=0x7fe9f415c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 shutdown_connections 2024-03-18T14:43:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.762+0000 7fe9f9a27640 1 -- 172.21.15.6:0/3765097818 wait complete. 2024-03-18T14:43:50.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:50.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.877+0000 7f8c3a577640 1 Processor -- start 2024-03-18T14:43:50.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.877+0000 7f8c3a577640 1 -- start start 2024-03-18T14:43:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.878+0000 7f8c3a577640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 0x7f8c340cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.878+0000 7f8c3a577640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c340d2970 con 0x7f8c340d2510 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.878+0000 7f8c3a577640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c340cfac0 con 0x7f8c340d0730 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.878+0000 7f8c3a577640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8c340cd8d0 con 0x7f8c340ce9d0 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.878+0000 7f8c337fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8c340d2510 0x7f8c340d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58124/0 (socket says 172.21.15.6:58124) 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.878+0000 7f8c337fe640 1 -- 172.21.15.6:0/1208352054 learned_addr learned my addr 172.21.15.6:0/1208352054 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778315624 0 0) 0x7f8c340d2970 con 0x7f8c340d2510 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c20003520 con 0x7f8c340d2510 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c33fff640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 0x7f8c340cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2715517167 0 0) 0x7f8c20003520 con 0x7f8c340d2510 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 msgr2=0x7f8c340cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 0x7f8c340cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 >> v1:172.21.15.92:6789/0 conn(0x7f8c340d0730 legacy=0x7f8c3415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c340cccc0 con 0x7f8c340d2510 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f8c24001670 con 0x7f8c340d2510 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 >> v1:172.21.15.6:6789/0 conn(0x7f8c340d2510 legacy=0x7f8c340d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 0x7f8c200039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c20003f20 0x7f8c20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20004d40 0x7f8c200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8c340cd8d0 con 0x7f8c20003f20 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2000bad0 con 0x7f8c20004d40 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8c2000bcd0 con 0x7f8c340ce9d0 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c337fe640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c20003f20 0x7f8c20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c33fff640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 0x7f8c200039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c38b14640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20004d40 0x7f8c200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c337fe640 1 -- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 msgr2=0x7f8c200039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.879+0000 7f8c337fe640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340ce9d0 0x7f8c200039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.880+0000 7f8c337fe640 1 -- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20004d40 msgr2=0x7f8c200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.880+0000 7f8c337fe640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20004d40 0x7f8c200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.880+0000 7f8c337fe640 1 -- 172.21.15.6:0/1208352054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c20003520 con 0x7f8c20003f20 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.880+0000 7f8c337fe640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c20003f20 0x7f8c20004800 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400c240 tx=0x7f8c20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00c3590a2f32f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.880+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24007d30 con 0x7f8c20003f20 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.880+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c24030070 con 0x7f8c20003f20 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.880+0000 7f8c32ffd640 1 -- 172.21.15.6:0/1208352054 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2402cd80 con 0x7f8c20003f20 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.881+0000 7f8c3a577640 1 -- 172.21.15.6:0/1208352054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c20003f20 msgr2=0x7f8c20004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.881+0000 7f8c3a577640 1 --2- 172.21.15.6:0/1208352054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c20003f20 0x7f8c20004800 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400c240 tx=0x7f8c20005830 comp rx=0 tx=0).stop 2024-03-18T14:43:50.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.881+0000 7f8c38b14640 1 -- 172.21.15.6:0/1208352054 reap_dead start 2024-03-18T14:43:50.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.881+0000 7f8c3a577640 1 -- 172.21.15.6:0/1208352054 shutdown_connections 2024-03-18T14:43:50.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.881+0000 7f8c3a577640 1 -- 172.21.15.6:0/1208352054 >> 172.21.15.6:0/1208352054 conn(0x7f8c3408b0c0 msgr2=0x7f8c340d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:50.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.881+0000 7f8c3a577640 1 -- 172.21.15.6:0/1208352054 shutdown_connections 2024-03-18T14:43:50.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.881+0000 7f8c3a577640 1 -- 172.21.15.6:0/1208352054 wait complete. 2024-03-18T14:43:50.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.882+0000 7f8c3a577640 1 Processor -- start 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.882+0000 7f8c3a577640 1 -- start start 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c3a577640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20003f20 0x7f8c340b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c3a577640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c3406d7f0 0x7f8c340ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c3a577640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340b6330 0x7f8c340b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c3a577640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8c340cd560 con 0x7f8c3406d7f0 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c3a577640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8c340cd280 con 0x7f8c20003f20 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c3a577640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f8c340cedb0 con 0x7f8c340b6330 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c337fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c3406d7f0 0x7f8c340ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c337fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c3406d7f0 0x7f8c340ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36194/0 (socket says 172.21.15.6:36194) 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c337fe640 1 -- 172.21.15.6:0/3241037474 learned_addr learned my addr 172.21.15.6:0/3241037474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c33fff640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20003f20 0x7f8c340b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c38b14640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340b6330 0x7f8c340b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c337fe640 1 -- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340b6330 msgr2=0x7f8c340b67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c337fe640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340b6330 0x7f8c340b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.883+0000 7f8c337fe640 1 -- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20003f20 msgr2=0x7f8c340b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c337fe640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20003f20 0x7f8c340b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c337fe640 1 -- 172.21.15.6:0/3241037474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c340b4e60 con 0x7f8c3406d7f0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c337fe640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c3406d7f0 0x7f8c340ca450 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400c240 tx=0x7f8c340cd840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c30ff9640 1 -- 172.21.15.6:0/3241037474 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24007d30 con 0x7f8c3406d7f0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c30ff9640 1 -- 172.21.15.6:0/3241037474 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c24004bf0 con 0x7f8c3406d7f0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c340b5140 con 0x7f8c3406d7f0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c30ff9640 1 -- 172.21.15.6:0/3241037474 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24007070 con 0x7f8c3406d7f0 2024-03-18T14:43:50.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.884+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c340b5700 con 0x7f8c3406d7f0 2024-03-18T14:43:50.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.885+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c340751b0 con 0x7f8c3406d7f0 2024-03-18T14:43:50.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.885+0000 7f8c30ff9640 1 -- 172.21.15.6:0/3241037474 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8c24047020 con 0x7f8c3406d7f0 2024-03-18T14:43:50.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.886+0000 7f8c30ff9640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c1406bc00 0x7f8c1406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:50.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.886+0000 7f8c30ff9640 1 -- 172.21.15.6:0/3241037474 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f8c240abe70 con 0x7f8c3406d7f0 2024-03-18T14:43:50.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.886+0000 7f8c33fff640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c1406bc00 0x7f8c1406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:50.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.886+0000 7f8c33fff640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c1406bc00 0x7f8c1406e0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0099f0 tx=0x7f8c1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:50.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:50.890+0000 7f8c30ff9640 1 -- 172.21.15.6:0/3241037474 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8c2407a740 con 0x7f8c3406d7f0 2024-03-18T14:43:51.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.128+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8c340d14e0 con 0x7f8c3406d7f0 2024-03-18T14:43:51.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.128+0000 7f8c30ff9640 1 -- 172.21.15.6:0/3241037474 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f8c2407a040 con 0x7f8c3406d7f0 2024-03-18T14:43:51.131 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c1406bc00 msgr2=0x7f8c1406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c1406bc00 0x7f8c1406e0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0099f0 tx=0x7f8c1c008040 comp rx=0 tx=0).stop 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c3406d7f0 msgr2=0x7f8c340ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c3406d7f0 0x7f8c340ca450 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400c240 tx=0x7f8c340cd840 comp rx=0 tx=0).stop 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 shutdown_connections 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f8c1406bc00 0x7f8c1406e0c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c340b6330 0x7f8c340b67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c3406d7f0 0x7f8c340ca450 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 --2- 172.21.15.6:0/3241037474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c20003f20 0x7f8c340b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 >> 172.21.15.6:0/3241037474 conn(0x7f8c3408b0c0 msgr2=0x7f8c340ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:51.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.130+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 shutdown_connections 2024-03-18T14:43:51.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.131+0000 7f8c3a577640 1 -- 172.21.15.6:0/3241037474 wait complete. 2024-03-18T14:43:51.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.247+0000 7fe7e63df640 1 Processor -- start 2024-03-18T14:43:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.247+0000 7fe7e63df640 1 -- start start 2024-03-18T14:43:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7e63df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e015cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7e63df640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe7e0071eb0 0x7fe7e00af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7e63df640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7e00afef0 con 0x7fe7e015f850 2024-03-18T14:43:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7e63df640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e00b01d0 con 0x7fe7e0071eb0 2024-03-18T14:43:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7e63df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e00b04b0 con 0x7fe7e0071ad0 2024-03-18T14:43:51.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e015cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7df7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe7e0071eb0 0x7fe7e00af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e015cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55458/0 (socket says 172.21.15.6:55458) 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.248+0000 7fe7dffff640 1 -- 172.21.15.6:0/1261209142 learned_addr learned my addr 172.21.15.6:0/1261209142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.249+0000 7fe7df7fe640 1 -- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 msgr2=0x7fe7e015cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.249+0000 7fe7df7fe640 1 --2- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e015cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.249+0000 7fe7df7fe640 1 -- 172.21.15.6:0/1261209142 >> v1:172.21.15.92:6789/0 conn(0x7fe7e015f850 legacy=0x7fe7e015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.249+0000 7fe7df7fe640 1 -- 172.21.15.6:0/1261209142 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e015d5a0 con 0x7fe7e0071eb0 2024-03-18T14:43:51.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.249+0000 7fe7dffff640 1 --2- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e015cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7df7fe640 1 --2- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 0x7fe7e00af580 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40097a0 tx=0x7fe7d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d501ab29ed15631f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7deffd640 1 -- 172.21.15.6:0/1261209142 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4035020 con 0x7fe7e0071eb0 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7deffd640 1 -- 172.21.15.6:0/1261209142 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d4031830 con 0x7fe7e0071eb0 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7deffd640 1 -- 172.21.15.6:0/1261209142 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4031c00 con 0x7fe7e0071eb0 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7e63df640 1 -- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 msgr2=0x7fe7e00af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7e63df640 1 --2- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 0x7fe7e00af580 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40097a0 tx=0x7fe7d402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7e63df640 1 -- 172.21.15.6:0/1261209142 shutdown_connections 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7e63df640 1 --2- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 0x7fe7e00af580 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7e63df640 1 --2- 172.21.15.6:0/1261209142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e015cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.250+0000 7fe7e63df640 1 -- 172.21.15.6:0/1261209142 >> 172.21.15.6:0/1261209142 conn(0x7fe7e006b1c0 msgr2=0x7fe7e006b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.251+0000 7fe7e63df640 1 -- 172.21.15.6:0/1261209142 shutdown_connections 2024-03-18T14:43:51.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.251+0000 7fe7e63df640 1 -- 172.21.15.6:0/1261209142 wait complete. 2024-03-18T14:43:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.252+0000 7fe7e63df640 1 Processor -- start 2024-03-18T14:43:51.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.252+0000 7fe7e63df640 1 -- start start 2024-03-18T14:43:51.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e63df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e0138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e63df640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 0x7fe7e00b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e63df640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e015f850 0x7fe7e0137320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e63df640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e0159be0 con 0x7fe7e015f850 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e63df640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e0159a60 con 0x7fe7e0071eb0 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e63df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e00afac0 con 0x7fe7e0071ad0 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e497c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e015f850 0x7fe7e0137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e497c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e015f850 0x7fe7e0137320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36198/0 (socket says 172.21.15.6:36198) 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e497c640 1 -- 172.21.15.6:0/2827758735 learned_addr learned my addr 172.21.15.6:0/2827758735 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7dffff640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e0138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7df7fe640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 0x7fe7e00b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7dffff640 1 -- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 msgr2=0x7fe7e00b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7dffff640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 0x7fe7e00b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7dffff640 1 -- 172.21.15.6:0/2827758735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e015f850 msgr2=0x7fe7e0137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7dffff640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e015f850 0x7fe7e0137320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7dffff640 1 -- 172.21.15.6:0/2827758735 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e00b4600 con 0x7fe7e0071ad0 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7e497c640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e015f850 0x7fe7e0137320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.254+0000 7fe7dffff640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e0138d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000e900 tx=0x7fe7d000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.255+0000 7fe7dcff9640 1 -- 172.21.15.6:0/2827758735 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d000ca00 con 0x7fe7e0071ad0 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.255+0000 7fe7dcff9640 1 -- 172.21.15.6:0/2827758735 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d000cba0 con 0x7fe7e0071ad0 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.255+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e0137cc0 con 0x7fe7e0071ad0 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.255+0000 7fe7dcff9640 1 -- 172.21.15.6:0/2827758735 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0004030 con 0x7fe7e0071ad0 2024-03-18T14:43:51.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.255+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e0138280 con 0x7fe7e0071ad0 2024-03-18T14:43:51.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.256+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e00751b0 con 0x7fe7e0071ad0 2024-03-18T14:43:51.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.256+0000 7fe7dcff9640 1 -- 172.21.15.6:0/2827758735 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe7d00041d0 con 0x7fe7e0071ad0 2024-03-18T14:43:51.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.257+0000 7fe7dcff9640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe7cc06bbd0 0x7fe7cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.257+0000 7fe7dcff9640 1 -- 172.21.15.6:0/2827758735 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fe7d008aeb0 con 0x7fe7e0071ad0 2024-03-18T14:43:51.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.257+0000 7fe7df7fe640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe7cc06bbd0 0x7fe7cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.257+0000 7fe7df7fe640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe7cc06bbd0 0x7fe7cc06e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4036000 tx=0x7fe7d4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.262+0000 7fe7dcff9640 1 -- 172.21.15.6:0/2827758735 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe7e00b4600 con 0x7fe7e0071ad0 2024-03-18T14:43:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.516+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe7e015b820 con 0x7fe7e0071ad0 2024-03-18T14:43:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.517+0000 7fe7dcff9640 1 -- 172.21.15.6:0/2827758735 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7fe7d0017020 con 0x7fe7e0071ad0 2024-03-18T14:43:51.519 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe7cc06bbd0 msgr2=0x7fe7cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe7cc06bbd0 0x7fe7cc06e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4036000 tx=0x7fe7d4008030 comp rx=0 tx=0).stop 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 msgr2=0x7fe7e0138d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e0138d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000e900 tx=0x7fe7d000edc0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 shutdown_connections 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe7cc06bbd0 0x7fe7cc06e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e015f850 0x7fe7e0137320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0071eb0 0x7fe7e00b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 --2- 172.21.15.6:0/2827758735 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e0071ad0 0x7fe7e0138d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 >> 172.21.15.6:0/2827758735 conn(0x7fe7e006b1c0 msgr2=0x7fe7e006f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 shutdown_connections 2024-03-18T14:43:51.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.519+0000 7fe7e63df640 1 -- 172.21.15.6:0/2827758735 wait complete. 2024-03-18T14:43:51.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.632+0000 7f19e9adc640 1 Processor -- start 2024-03-18T14:43:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.632+0000 7f19e9adc640 1 -- start start 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e9adc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e40d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e9adc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19e40b6df0 0x7f19e40b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e9adc640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19e40d1f30 con 0x7f19e406cb80 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e9adc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f19e40b9cf0 con 0x7f19e40b6df0 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e9adc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f19e40d2210 con 0x7f19e406c7a0 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19e40b6df0 0x7f19e40b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e3fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19e40b6df0 0x7f19e40b7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36208/0 (socket says 172.21.15.6:36208) 2024-03-18T14:43:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.633+0000 7f19e3fff640 1 -- 172.21.15.6:0/1928886855 learned_addr learned my addr 172.21.15.6:0/1928886855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.634+0000 7f19e37fe640 1 --2- 172.21.15.6:0/1928886855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e40d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.634+0000 7f19e3fff640 1 -- 172.21.15.6:0/1928886855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 msgr2=0x7f19e40d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.634+0000 7f19e3fff640 1 --2- 172.21.15.6:0/1928886855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e40d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.634+0000 7f19e3fff640 1 -- 172.21.15.6:0/1928886855 >> v1:172.21.15.92:6789/0 conn(0x7f19e406cb80 legacy=0x7f19e40b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.634+0000 7f19e3fff640 1 -- 172.21.15.6:0/1928886855 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e40d24f0 con 0x7f19e40b6df0 2024-03-18T14:43:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.634+0000 7f19e3fff640 1 --2- 172.21.15.6:0/1928886855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e40b6df0 0x7f19e40b7250 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009e50 tx=0x7f19dc02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4867e3981b296e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e27fc640 1 -- 172.21.15.6:0/1928886855 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc02cb90 con 0x7f19e40b6df0 2024-03-18T14:43:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e27fc640 1 -- 172.21.15.6:0/1928886855 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19dc02cd30 con 0x7f19e40b6df0 2024-03-18T14:43:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e27fc640 1 -- 172.21.15.6:0/1928886855 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc034610 con 0x7f19e40b6df0 2024-03-18T14:43:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 -- 172.21.15.6:0/1928886855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e40b6df0 msgr2=0x7f19e40b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 --2- 172.21.15.6:0/1928886855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e40b6df0 0x7f19e40b7250 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009e50 tx=0x7f19dc02bff0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 -- 172.21.15.6:0/1928886855 shutdown_connections 2024-03-18T14:43:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 --2- 172.21.15.6:0/1928886855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e40b6df0 0x7f19e40b7250 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 --2- 172.21.15.6:0/1928886855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e40d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 -- 172.21.15.6:0/1928886855 >> 172.21.15.6:0/1928886855 conn(0x7f19e4072ce0 msgr2=0x7f19e40730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 -- 172.21.15.6:0/1928886855 shutdown_connections 2024-03-18T14:43:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.635+0000 7f19e9adc640 1 -- 172.21.15.6:0/1928886855 wait complete. 2024-03-18T14:43:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.637+0000 7f19e9adc640 1 Processor -- start 2024-03-18T14:43:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.637+0000 7f19e9adc640 1 -- start start 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e9adc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e41548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e9adc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e406cb80 0x7f19e40b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e9adc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19e40b6df0 0x7f19e41565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e9adc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f19e406cfe0 con 0x7f19e406cb80 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e9adc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19e40d2ab0 con 0x7f19e40b6df0 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e9adc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f19e40d37f0 con 0x7f19e406c7a0 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e2ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e406cb80 0x7f19e40b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.638+0000 7f19e2ffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e406cb80 0x7f19e40b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36218/0 (socket says 172.21.15.6:36218) 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e2ffd640 1 -- 172.21.15.6:0/2230867639 learned_addr learned my addr 172.21.15.6:0/2230867639 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e3fff640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19e40b6df0 0x7f19e41565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e37fe640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e41548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e2ffd640 1 -- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 msgr2=0x7f19e41548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e2ffd640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e41548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e2ffd640 1 -- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19e40b6df0 msgr2=0x7f19e41565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e2ffd640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19e40b6df0 0x7f19e41565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e2ffd640 1 -- 172.21.15.6:0/2230867639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e4154ed0 con 0x7f19e406cb80 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e3fff640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19e40b6df0 0x7f19e41565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e2ffd640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e406cb80 0x7f19e40b4cc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00c430 tx=0x7f19cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19dbfff640 1 -- 172.21.15.6:0/2230867639 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0052d0 con 0x7f19e406cb80 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19dbfff640 1 -- 172.21.15.6:0/2230867639 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19cc005490 con 0x7f19e406cb80 2024-03-18T14:43:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e415f1e0 con 0x7f19e406cb80 2024-03-18T14:43:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19dbfff640 1 -- 172.21.15.6:0/2230867639 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc007260 con 0x7f19e406cb80 2024-03-18T14:43:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.639+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e415f7a0 con 0x7f19e406cb80 2024-03-18T14:43:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.640+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e404f5b0 con 0x7f19e406cb80 2024-03-18T14:43:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.641+0000 7f19dbfff640 1 -- 172.21.15.6:0/2230867639 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f19cc007400 con 0x7f19e406cb80 2024-03-18T14:43:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.641+0000 7f19dbfff640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f19c406bbd0 0x7f19c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.641+0000 7f19dbfff640 1 -- 172.21.15.6:0/2230867639 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f19cc08a100 con 0x7f19e406cb80 2024-03-18T14:43:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.641+0000 7f19e37fe640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f19c406bbd0 0x7f19c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.642+0000 7f19e37fe640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f19c406bbd0 0x7f19c406e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f19d4006fd0 tx=0x7f19d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.646+0000 7f19dbfff640 1 -- 172.21.15.6:0/2230867639 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f19cc058a60 con 0x7f19e406cb80 2024-03-18T14:43:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.880+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f19e40b4aa0 con 0x7f19e406cb80 2024-03-18T14:43:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.880+0000 7f19dbfff640 1 -- 172.21.15.6:0/2230867639 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f19cc058360 con 0x7f19e406cb80 2024-03-18T14:43:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f19c406bbd0 msgr2=0x7f19c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f19c406bbd0 0x7f19c406e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f19d4006fd0 tx=0x7f19d4008040 comp rx=0 tx=0).stop 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e406cb80 msgr2=0x7f19e40b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e406cb80 0x7f19e40b4cc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00c430 tx=0x7f19cc00c8f0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 shutdown_connections 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f19c406bbd0 0x7f19c406e090 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19e40b6df0 0x7f19e41565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19e406cb80 0x7f19e40b4cc0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 --2- 172.21.15.6:0/2230867639 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19e406c7a0 0x7f19e41548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 >> 172.21.15.6:0/2230867639 conn(0x7f19e4072ce0 msgr2=0x7f19e40b91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 shutdown_connections 2024-03-18T14:43:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:51.882+0000 7f19e9adc640 1 -- 172.21.15.6:0/2230867639 wait complete. 2024-03-18T14:43:51.895 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:51.895 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:51.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:52.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.050+0000 7f5ada0ac640 1 Processor -- start 2024-03-18T14:43:52.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.050+0000 7f5ada0ac640 1 -- start start 2024-03-18T14:43:52.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ada0ac640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad40d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ada0ac640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5ad40d1d90 0x7f5ad40cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ada0ac640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad40cc870 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ada0ac640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5ad40d0500 con 0x7f5ad40d1d90 2024-03-18T14:43:52.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ada0ac640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad40d07e0 con 0x7f5ad415bd40 2024-03-18T14:43:52.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad2ffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5ad40d1d90 0x7f5ad40cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ad40cfcd0 0x7f5ad40ce540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58128/0 (socket says 172.21.15.6:58128) 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad3fff640 1 -- 172.21.15.6:0/1708019181 learned_addr learned my addr 172.21.15.6:0/1708019181 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad37fe640 1 --2- 172.21.15.6:0/1708019181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad40d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad27fc640 1 -- 172.21.15.6:0/1708019181 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747634683 0 0) 0x7f5ad40cc870 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad27fc640 1 -- 172.21.15.6:0/1708019181 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5abc003520 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad2ffd640 1 -- 172.21.15.6:0/1708019181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 msgr2=0x7f5ad40d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad2ffd640 1 --2- 172.21.15.6:0/1708019181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad40d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad2ffd640 1 -- 172.21.15.6:0/1708019181 >> v1:172.21.15.6:6789/0 conn(0x7f5ad40cfcd0 legacy=0x7f5ad40ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.051+0000 7f5ad2ffd640 1 -- 172.21.15.6:0/1708019181 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad40ccb10 con 0x7f5ad40d1d90 2024-03-18T14:43:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.052+0000 7f5ad2ffd640 1 --2- 172.21.15.6:0/1708019181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40cf790 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40097a0 tx=0x7f5ac402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44338eb718f03274 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.052+0000 7f5ad27fc640 1 -- 172.21.15.6:0/1708019181 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4035020 con 0x7f5ad40d1d90 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.052+0000 7f5ad27fc640 1 -- 172.21.15.6:0/1708019181 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac4031830 con 0x7f5ad40d1d90 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.052+0000 7f5ad27fc640 1 -- 172.21.15.6:0/1708019181 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4031c00 con 0x7f5ad40d1d90 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 -- 172.21.15.6:0/1708019181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40d1d90 msgr2=0x7f5ad40cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/1708019181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40cf790 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40097a0 tx=0x7f5ac402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 -- 172.21.15.6:0/1708019181 shutdown_connections 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/1708019181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40cf790 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/1708019181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad40d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 -- 172.21.15.6:0/1708019181 >> 172.21.15.6:0/1708019181 conn(0x7f5ad409ae20 msgr2=0x7f5ad408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 -- 172.21.15.6:0/1708019181 shutdown_connections 2024-03-18T14:43:52.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.053+0000 7f5ada0ac640 1 -- 172.21.15.6:0/1708019181 wait complete. 2024-03-18T14:43:52.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.054+0000 7f5ada0ac640 1 Processor -- start 2024-03-18T14:43:52.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.054+0000 7f5ada0ac640 1 -- start start 2024-03-18T14:43:52.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ada0ac640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40cfcd0 0x7f5ad40ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ada0ac640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ada0ac640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad41f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ada0ac640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad40d0c40 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ada0ac640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad40d0ac0 con 0x7f5ad40d1d90 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ada0ac640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad40d0160 con 0x7f5ad415bd40 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40cfcd0 0x7f5ad40ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40cfcd0 0x7f5ad40ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36226/0 (socket says 172.21.15.6:36226) 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 -- 172.21.15.6:0/3858545290 learned_addr learned my addr 172.21.15.6:0/3858545290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad2ffd640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad3fff640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad41f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 -- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 msgr2=0x7f5ad41f0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad41f0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 -- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ad40d1d90 msgr2=0x7f5ad40c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad37fe640 1 -- 172.21.15.6:0/3858545290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad40ca8b0 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.055+0000 7f5ad2ffd640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.056+0000 7f5ad37fe640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40cfcd0 0x7f5ad40ce140 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e8a0 tx=0x7f5ac800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.056+0000 7f5abbfff640 1 -- 172.21.15.6:0/3858545290 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8017070 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.056+0000 7f5abbfff640 1 -- 172.21.15.6:0/3858545290 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac800c940 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.056+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad40cab90 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.056+0000 7f5abbfff640 1 -- 172.21.15.6:0/3858545290 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800cd30 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.056+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad40cb150 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.057+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad40cb590 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.057+0000 7f5abbfff640 1 -- 172.21.15.6:0/3858545290 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5ac801a070 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.058+0000 7f5abbfff640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5ac006bbd0 0x7f5ac006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.058+0000 7f5abbfff640 1 -- 172.21.15.6:0/3858545290 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f5ac808adc0 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.058+0000 7f5ad2ffd640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5ac006bbd0 0x7f5ac006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.059+0000 7f5ad2ffd640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5ac006bbd0 0x7f5ac006e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4036000 tx=0x7f5ac4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.062+0000 7f5abbfff640 1 -- 172.21.15.6:0/3858545290 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5ac80597b0 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.248+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ad404f620 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.249+0000 7f5abbfff640 1 -- 172.21.15.6:0/3858545290 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7f5ac80590b0 con 0x7f5ad40cfcd0 2024-03-18T14:43:52.251 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:52.252 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:52.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5ac006bbd0 msgr2=0x7f5ac006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5ac006bbd0 0x7f5ac006e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4036000 tx=0x7f5ac4008030 comp rx=0 tx=0).stop 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40cfcd0 msgr2=0x7f5ad40ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40cfcd0 0x7f5ad40ce140 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e8a0 tx=0x7f5ac800ed60 comp rx=0 tx=0).stop 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 shutdown_connections 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5ac006bbd0 0x7f5ac006e090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ad415bd40 0x7f5ad41f0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ad40d1d90 0x7f5ad40c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 --2- 172.21.15.6:0/3858545290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ad40cfcd0 0x7f5ad40ce140 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 >> 172.21.15.6:0/3858545290 conn(0x7f5ad409ae20 msgr2=0x7f5ad415ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 shutdown_connections 2024-03-18T14:43:52.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.251+0000 7f5ada0ac640 1 -- 172.21.15.6:0/3858545290 wait complete. 2024-03-18T14:43:52.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:43:39.597879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:43:40.299491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:52.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:52.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.367+0000 7fb1eb427640 1 Processor -- start 2024-03-18T14:43:52.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.367+0000 7fb1eb427640 1 -- start start 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1eb427640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1e40ce930 0x7fb1e406cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1eb427640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb1e406d020 0x7fb1e40d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1eb427640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40d0680 0x7fb1e415d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1eb427640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e40d0f30 con 0x7fb1e40ce930 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1eb427640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e40b1070 con 0x7fb1e406d020 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1eb427640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e40b1350 con 0x7fb1e40d0680 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1e40ce930 0x7fb1e406cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb1e40ce930 0x7fb1e406cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36234/0 (socket says 172.21.15.6:36234) 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 -- 172.21.15.6:0/1158156703 learned_addr learned my addr 172.21.15.6:0/1158156703 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:52.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e99c4640 1 --2- 172.21.15.6:0/1158156703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40d0680 0x7fb1e415d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e89c2640 1 --2- 172.21.15.6:0/1158156703 >> v2:172.21.15.92:3300/0 conn(0x7fb1e406d020 0x7fb1e40d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 -- 172.21.15.6:0/1158156703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40d0680 msgr2=0x7fb1e415d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:52.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 --2- 172.21.15.6:0/1158156703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40d0680 0x7fb1e415d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 -- 172.21.15.6:0/1158156703 >> v2:172.21.15.92:3300/0 conn(0x7fb1e406d020 msgr2=0x7fb1e40d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:52.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 --2- 172.21.15.6:0/1158156703 >> v2:172.21.15.92:3300/0 conn(0x7fb1e406d020 0x7fb1e40d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e91c3640 1 -- 172.21.15.6:0/1158156703 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e40d0b10 con 0x7fb1e40ce930 2024-03-18T14:43:52.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.368+0000 7fb1e99c4640 1 --2- 172.21.15.6:0/1158156703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40d0680 0x7fb1e415d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.369+0000 7fb1e91c3640 1 --2- 172.21.15.6:0/1158156703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e40ce930 0x7fb1e406cae0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4072300 tx=0x7fb1e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bb78811edcdf0c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.369+0000 7fb1dbfff640 1 -- 172.21.15.6:0/1158156703 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e002d040 con 0x7fb1e40ce930 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.369+0000 7fb1dbfff640 1 -- 172.21.15.6:0/1158156703 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e0004230 con 0x7fb1e40ce930 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.369+0000 7fb1dbfff640 1 -- 172.21.15.6:0/1158156703 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0003610 con 0x7fb1e40ce930 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 -- 172.21.15.6:0/1158156703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e40ce930 msgr2=0x7fb1e406cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 --2- 172.21.15.6:0/1158156703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e40ce930 0x7fb1e406cae0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4072300 tx=0x7fb1e002b8f0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 -- 172.21.15.6:0/1158156703 shutdown_connections 2024-03-18T14:43:52.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 --2- 172.21.15.6:0/1158156703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40d0680 0x7fb1e415d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 --2- 172.21.15.6:0/1158156703 >> v2:172.21.15.92:3300/0 conn(0x7fb1e406d020 0x7fb1e40d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 --2- 172.21.15.6:0/1158156703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e40ce930 0x7fb1e406cae0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 -- 172.21.15.6:0/1158156703 >> 172.21.15.6:0/1158156703 conn(0x7fb1e408b0c0 msgr2=0x7fb1e40d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 -- 172.21.15.6:0/1158156703 shutdown_connections 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.370+0000 7fb1eb427640 1 -- 172.21.15.6:0/1158156703 wait complete. 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.371+0000 7fb1eb427640 1 Processor -- start 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.371+0000 7fb1eb427640 1 -- start start 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1eb427640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e406d020 0x7fb1e4148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1eb427640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40ce930 0x7fb1e41f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1eb427640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1e40d0680 0x7fb1e40c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1eb427640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e415dd70 con 0x7fb1e406d020 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1eb427640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e415dbf0 con 0x7fb1e40d0680 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1eb427640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e415def0 con 0x7fb1e40ce930 2024-03-18T14:43:52.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e406d020 0x7fb1e4148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e406d020 0x7fb1e4148e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36244/0 (socket says 172.21.15.6:36244) 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 -- 172.21.15.6:0/2601551377 learned_addr learned my addr 172.21.15.6:0/2601551377 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e99c4640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1e40d0680 0x7fb1e40c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e89c2640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40ce930 0x7fb1e41f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 -- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40ce930 msgr2=0x7fb1e41f3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40ce930 0x7fb1e41f3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 -- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1e40d0680 msgr2=0x7fb1e40c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1e40d0680 0x7fb1e40c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 -- 172.21.15.6:0/2601551377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e4149390 con 0x7fb1e406d020 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e99c4640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1e40d0680 0x7fb1e40c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e89c2640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40ce930 0x7fb1e41f3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.372+0000 7fb1e91c3640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e406d020 0x7fb1e4148e50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00090b0 tx=0x7fb1e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.373+0000 7fb1d9ffb640 1 -- 172.21.15.6:0/2601551377 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0003c50 con 0x7fb1e406d020 2024-03-18T14:43:52.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.373+0000 7fb1d9ffb640 1 -- 172.21.15.6:0/2601551377 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e0003e10 con 0x7fb1e406d020 2024-03-18T14:43:52.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.373+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e40c6e50 con 0x7fb1e406d020 2024-03-18T14:43:52.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.373+0000 7fb1d9ffb640 1 -- 172.21.15.6:0/2601551377 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0006280 con 0x7fb1e406d020 2024-03-18T14:43:52.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.373+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e414ac40 con 0x7fb1e406d020 2024-03-18T14:43:52.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.374+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e40751b0 con 0x7fb1e406d020 2024-03-18T14:43:52.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.375+0000 7fb1d9ffb640 1 -- 172.21.15.6:0/2601551377 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb1e0006420 con 0x7fb1e406d020 2024-03-18T14:43:52.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.375+0000 7fb1d9ffb640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb1c806bbd0 0x7fb1c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.375+0000 7fb1d9ffb640 1 -- 172.21.15.6:0/2601551377 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fb1e00acda0 con 0x7fb1e406d020 2024-03-18T14:43:52.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.376+0000 7fb1e89c2640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb1c806bbd0 0x7fb1c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.376+0000 7fb1e89c2640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb1c806bbd0 0x7fb1c806e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40b1040 tx=0x7fb1d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.380+0000 7fb1d9ffb640 1 -- 172.21.15.6:0/2601551377 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb1e4149390 con 0x7fb1e406d020 2024-03-18T14:43:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.562+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb1e408b680 con 0x7fb1e406d020 2024-03-18T14:43:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.562+0000 7fb1d9ffb640 1 -- 172.21.15.6:0/2601551377 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fb1e007b700 con 0x7fb1e406d020 2024-03-18T14:43:52.565 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb1c806bbd0 msgr2=0x7fb1c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb1c806bbd0 0x7fb1c806e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40b1040 tx=0x7fb1d4008040 comp rx=0 tx=0).stop 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e406d020 msgr2=0x7fb1e4148e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e406d020 0x7fb1e4148e50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00090b0 tx=0x7fb1e00046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 shutdown_connections 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb1c806bbd0 0x7fb1c806e090 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1e40d0680 0x7fb1e40c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb1e40ce930 0x7fb1e41f3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 --2- 172.21.15.6:0/2601551377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1e406d020 0x7fb1e4148e50 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 >> 172.21.15.6:0/2601551377 conn(0x7fb1e408b0c0 msgr2=0x7fb1e4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 shutdown_connections 2024-03-18T14:43:52.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.564+0000 7fb1eb427640 1 -- 172.21.15.6:0/2601551377 wait complete. 2024-03-18T14:43:52.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:52.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.682+0000 7fadd184d640 1 Processor -- start 2024-03-18T14:43:52.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.682+0000 7fadd184d640 1 -- start start 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadd184d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc15e030 0x7fadcc0d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadd184d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadcc0b65a0 con 0x7fadcc0d0450 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadd184d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadcc0d2fd0 con 0x7fadcc0b6140 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadd184d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fadcc15e410 con 0x7fadcc15e030 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadcb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fadcc0d0450 0x7fadcc0d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58130/0 (socket says 172.21.15.6:58130) 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadcb7fe640 1 -- 172.21.15.6:0/2910440397 learned_addr learned my addr 172.21.15.6:0/2910440397 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadcaffd640 1 --2- 172.21.15.6:0/2910440397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc15e030 0x7fadcc0d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadc9ffb640 1 -- 172.21.15.6:0/2910440397 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3689439844 0 0) 0x7fadcc0b65a0 con 0x7fadcc0d0450 2024-03-18T14:43:52.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.683+0000 7fadc9ffb640 1 -- 172.21.15.6:0/2910440397 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadb4003520 con 0x7fadcc0d0450 2024-03-18T14:43:52.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.684+0000 7fadc9ffb640 1 -- 172.21.15.6:0/2910440397 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 499102372 0 0) 0x7fadb4003520 con 0x7fadcc0d0450 2024-03-18T14:43:52.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.684+0000 7fadcaffd640 1 -- 172.21.15.6:0/2910440397 >> v1:172.21.15.92:6789/0 conn(0x7fadcc0b6140 legacy=0x7fadcc0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.684+0000 7fadcaffd640 1 -- 172.21.15.6:0/2910440397 >> v1:172.21.15.6:6789/0 conn(0x7fadcc0d0450 legacy=0x7fadcc0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.684+0000 7fadcaffd640 1 -- 172.21.15.6:0/2910440397 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadcc0d3290 con 0x7fadcc15e030 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadcaffd640 1 --2- 172.21.15.6:0/2910440397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc15e030 0x7fadcc0d3570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0097d0 tx=0x7fadbc02b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bd98be6ff7a560d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadc9ffb640 1 -- 172.21.15.6:0/2910440397 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc037070 con 0x7fadcc15e030 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadc9ffb640 1 -- 172.21.15.6:0/2910440397 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadbc030510 con 0x7fadcc15e030 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadc9ffb640 1 -- 172.21.15.6:0/2910440397 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc02be60 con 0x7fadcc15e030 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadd184d640 1 -- 172.21.15.6:0/2910440397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc15e030 msgr2=0x7fadcc0d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadd184d640 1 --2- 172.21.15.6:0/2910440397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc15e030 0x7fadcc0d3570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0097d0 tx=0x7fadbc02b660 comp rx=0 tx=0).stop 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadd184d640 1 -- 172.21.15.6:0/2910440397 shutdown_connections 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadd184d640 1 --2- 172.21.15.6:0/2910440397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc15e030 0x7fadcc0d3570 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.685+0000 7fadd184d640 1 -- 172.21.15.6:0/2910440397 >> 172.21.15.6:0/2910440397 conn(0x7fadcc06c8c0 msgr2=0x7fadcc06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.686+0000 7fadd184d640 1 -- 172.21.15.6:0/2910440397 shutdown_connections 2024-03-18T14:43:52.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.686+0000 7fadd184d640 1 -- 172.21.15.6:0/2910440397 wait complete. 2024-03-18T14:43:52.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.687+0000 7fadd184d640 1 Processor -- start 2024-03-18T14:43:52.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.687+0000 7fadd184d640 1 -- start start 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.687+0000 7fadd184d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadcc0b6140 0x7fadcc158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadd184d640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc0d0450 0x7fadcc0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadd184d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadcc15e030 0x7fadcc0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadd184d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc0ce830 con 0x7fadcc15e030 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadd184d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc0ce300 con 0x7fadcc0b6140 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadd184d640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fadcc161a90 con 0x7fadcc0d0450 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadcc15e030 0x7fadcc0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadcc15e030 0x7fadcc0c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36256/0 (socket says 172.21.15.6:36256) 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 -- 172.21.15.6:0/3517169027 learned_addr learned my addr 172.21.15.6:0/3517169027 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcaffd640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadcc0b6140 0x7fadcc158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadca7fc640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc0d0450 0x7fadcc0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 -- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc0d0450 msgr2=0x7fadcc0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc0d0450 0x7fadcc0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 -- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadcc0b6140 msgr2=0x7fadcc158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadcc0b6140 0x7fadcc158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 -- 172.21.15.6:0/3517169027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadcc0c8cf0 con 0x7fadcc15e030 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadcb7fe640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadcc15e030 0x7fadcc0c8080 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fadc400cac0 tx=0x7fadc400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadb3fff640 1 -- 172.21.15.6:0/3517169027 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4018070 con 0x7fadcc15e030 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadb3fff640 1 -- 172.21.15.6:0/3517169027 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc400a730 con 0x7fadcc15e030 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadb3fff640 1 -- 172.21.15.6:0/3517169027 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc400ab20 con 0x7fadcc15e030 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.688+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadcc0c8fd0 con 0x7fadcc15e030 2024-03-18T14:43:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.689+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadcc0cbe70 con 0x7fadcc15e030 2024-03-18T14:43:52.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.689+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadcc067b50 con 0x7fadcc15e030 2024-03-18T14:43:52.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.690+0000 7fadb3fff640 1 -- 172.21.15.6:0/3517169027 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fadc400acc0 con 0x7fadcc15e030 2024-03-18T14:43:52.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.690+0000 7fadb3fff640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fadb806bc00 0x7fadb806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:52.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.690+0000 7fadb3fff640 1 -- 172.21.15.6:0/3517169027 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fadc408a370 con 0x7fadcc15e030 2024-03-18T14:43:52.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.690+0000 7fadcaffd640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fadb806bc00 0x7fadb806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:52.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.691+0000 7fadcaffd640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fadb806bc00 0x7fadb806e0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fadbc034000 tx=0x7fadbc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:52.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.695+0000 7fadb3fff640 1 -- 172.21.15.6:0/3517169027 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fadcc067b50 con 0x7fadcc15e030 2024-03-18T14:43:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.875+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fadcc0751b0 con 0x7fadcc15e030 2024-03-18T14:43:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.875+0000 7fadb3fff640 1 -- 172.21.15.6:0/3517169027 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7fadc4058680 con 0x7fadcc15e030 2024-03-18T14:43:52.878 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fadb806bc00 msgr2=0x7fadb806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fadb806bc00 0x7fadb806e0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fadbc034000 tx=0x7fadbc008030 comp rx=0 tx=0).stop 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadcc15e030 msgr2=0x7fadcc0c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadcc15e030 0x7fadcc0c8080 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fadc400cac0 tx=0x7fadc400cf80 comp rx=0 tx=0).stop 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 shutdown_connections 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fadb806bc00 0x7fadb806e0c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadcc15e030 0x7fadcc0c8080 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadcc0d0450 0x7fadcc0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 --2- 172.21.15.6:0/3517169027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadcc0b6140 0x7fadcc158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 >> 172.21.15.6:0/3517169027 conn(0x7fadcc06c8c0 msgr2=0x7fadcc0b7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 shutdown_connections 2024-03-18T14:43:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:52.877+0000 7fadd184d640 1 -- 172.21.15.6:0/3517169027 wait complete. 2024-03-18T14:43:52.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:53.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.014+0000 7f0c00287640 1 Processor -- start 2024-03-18T14:43:53.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.014+0000 7f0c00287640 1 -- start start 2024-03-18T14:43:53.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0c00287640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bf80ce9d0 0x7f0bf80cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0c00287640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80d0730 0x7f0bf815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0c00287640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf80d29a0 con 0x7f0bf80d2510 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0c00287640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf80cd8d0 con 0x7f0bf80ce9d0 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0c00287640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf80cfac0 con 0x7f0bf80d0730 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe023640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bf80ce9d0 0x7f0bf80cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe023640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bf80ce9d0 0x7f0bf80cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36266/0 (socket says 172.21.15.6:36266) 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe023640 1 -- 172.21.15.6:0/4232250391 learned_addr learned my addr 172.21.15.6:0/4232250391 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe824640 1 --2- 172.21.15.6:0/4232250391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80d0730 0x7f0bf815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe023640 1 -- 172.21.15.6:0/4232250391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80d0730 msgr2=0x7f0bf815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe023640 1 --2- 172.21.15.6:0/4232250391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80d0730 0x7f0bf815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe023640 1 -- 172.21.15.6:0/4232250391 >> v1:172.21.15.92:6789/0 conn(0x7f0bf80d2510 legacy=0x7f0bf80d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe023640 1 -- 172.21.15.6:0/4232250391 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf80cccc0 con 0x7f0bf80ce9d0 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.015+0000 7f0bfe824640 1 --2- 172.21.15.6:0/4232250391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80d0730 0x7f0bf815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.016+0000 7f0bfe023640 1 --2- 172.21.15.6:0/4232250391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80ce9d0 0x7f0bf80cc650 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009850 tx=0x7f0bf402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34017ef04c647baa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.016+0000 7f0bfd021640 1 -- 172.21.15.6:0/4232250391 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4004230 con 0x7f0bf80ce9d0 2024-03-18T14:43:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.016+0000 7f0bfd021640 1 -- 172.21.15.6:0/4232250391 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf40043d0 con 0x7f0bf80ce9d0 2024-03-18T14:43:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.016+0000 7f0bfd021640 1 -- 172.21.15.6:0/4232250391 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4031830 con 0x7f0bf80ce9d0 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 -- 172.21.15.6:0/4232250391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80ce9d0 msgr2=0x7f0bf80cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 --2- 172.21.15.6:0/4232250391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80ce9d0 0x7f0bf80cc650 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009850 tx=0x7f0bf402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 -- 172.21.15.6:0/4232250391 shutdown_connections 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 --2- 172.21.15.6:0/4232250391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80d0730 0x7f0bf815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 --2- 172.21.15.6:0/4232250391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80ce9d0 0x7f0bf80cc650 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 -- 172.21.15.6:0/4232250391 >> 172.21.15.6:0/4232250391 conn(0x7f0bf808b0c0 msgr2=0x7f0bf80d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 -- 172.21.15.6:0/4232250391 shutdown_connections 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.017+0000 7f0c00287640 1 -- 172.21.15.6:0/4232250391 wait complete. 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.019+0000 7f0c00287640 1 Processor -- start 2024-03-18T14:43:53.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.019+0000 7f0c00287640 1 -- start start 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0c00287640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80ce9d0 0x7f0bf815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0c00287640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf80d0730 0x7f0bf80b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0c00287640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80d2510 0x7f0bf80b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0c00287640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf80cd560 con 0x7f0bf80d2510 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0c00287640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf80cd280 con 0x7f0bf80d0730 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0c00287640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf80cedb0 con 0x7f0bf80ce9d0 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0bfe824640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80d2510 0x7f0bf80b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0bfe824640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80d2510 0x7f0bf80b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36280/0 (socket says 172.21.15.6:36280) 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0bfe824640 1 -- 172.21.15.6:0/2174054685 learned_addr learned my addr 172.21.15.6:0/2174054685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0bfe023640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80ce9d0 0x7f0bf815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.020+0000 7f0bfd822640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf80d0730 0x7f0bf80b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0bfe824640 1 -- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80ce9d0 msgr2=0x7f0bf815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0bfe824640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80ce9d0 0x7f0bf815dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0bfe824640 1 -- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf80d0730 msgr2=0x7f0bf80b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0bfe824640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf80d0730 0x7f0bf80b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0bfe824640 1 -- 172.21.15.6:0/2174054685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf81592c0 con 0x7f0bf80d2510 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0bfe824640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80d2510 0x7f0bf80b6d60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0007e40 tx=0x7f0bf000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0beeffd640 1 -- 172.21.15.6:0/2174054685 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf001a070 con 0x7f0bf80d2510 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0beeffd640 1 -- 172.21.15.6:0/2174054685 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf0015600 con 0x7f0bf80d2510 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf80b7390 con 0x7f0bf80d2510 2024-03-18T14:43:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0beeffd640 1 -- 172.21.15.6:0/2174054685 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf00159f0 con 0x7f0bf80d2510 2024-03-18T14:43:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.021+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf81f2a50 con 0x7f0bf80d2510 2024-03-18T14:43:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.022+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf80751b0 con 0x7f0bf80d2510 2024-03-18T14:43:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.022+0000 7f0beeffd640 1 -- 172.21.15.6:0/2174054685 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0bf001d070 con 0x7f0bf80d2510 2024-03-18T14:43:53.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.023+0000 7f0beeffd640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0bdc06bbd0 0x7f0bdc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.023+0000 7f0beeffd640 1 -- 172.21.15.6:0/2174054685 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f0bf008ae90 con 0x7f0bf80d2510 2024-03-18T14:43:53.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.023+0000 7f0bfe023640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0bdc06bbd0 0x7f0bdc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.024+0000 7f0bfe023640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0bdc06bbd0 0x7f0bdc06e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4035000 tx=0x7f0bf4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.027+0000 7f0beeffd640 1 -- 172.21.15.6:0/2174054685 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0bf0010070 con 0x7f0bf80d2510 2024-03-18T14:43:53.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.209+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0bf80cfe00 con 0x7f0bf80d2510 2024-03-18T14:43:53.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.210+0000 7f0beeffd640 1 -- 172.21.15.6:0/2174054685 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f0bf0059880 con 0x7f0bf80d2510 2024-03-18T14:43:53.212 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.211+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0bdc06bbd0 msgr2=0x7f0bdc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.211+0000 7f0c00287640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0bdc06bbd0 0x7f0bdc06e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4035000 tx=0x7f0bf4008030 comp rx=0 tx=0).stop 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.211+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80d2510 msgr2=0x7f0bf80b6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.211+0000 7f0c00287640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80d2510 0x7f0bf80b6d60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0007e40 tx=0x7f0bf000d4a0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 shutdown_connections 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0bdc06bbd0 0x7f0bdc06e090 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80d2510 0x7f0bf80b6d60 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf80d0730 0x7f0bf80b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 --2- 172.21.15.6:0/2174054685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf80ce9d0 0x7f0bf815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 >> 172.21.15.6:0/2174054685 conn(0x7f0bf808b0c0 msgr2=0x7f0bf8070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 shutdown_connections 2024-03-18T14:43:53.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.212+0000 7f0c00287640 1 -- 172.21.15.6:0/2174054685 wait complete. 2024-03-18T14:43:53.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.332+0000 7fd5ec6ff640 1 Processor -- start 2024-03-18T14:43:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.332+0000 7fd5ec6ff640 1 -- start start 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5ec6ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5e4072900 0x7fd5e40b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5ec6ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e40b40d0 0x7fd5e40cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5ec6ff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e406da60 con 0x7fd5e415a440 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5ec6ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd5e406dd40 con 0x7fd5e4072900 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5ec6ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e406e020 con 0x7fd5e40b40d0 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9c9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5e4072900 0x7fd5e40b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9c9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5e4072900 0x7fd5e40b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36290/0 (socket says 172.21.15.6:36290) 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9c9a640 1 -- 172.21.15.6:0/3314411212 learned_addr learned my addr 172.21.15.6:0/3314411212 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:53.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5ea49b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd5e415a440 0x7fd5e40721f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58142/0 (socket says 172.21.15.6:58142) 2024-03-18T14:43:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5eac9c640 1 --2- 172.21.15.6:0/3314411212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e40b40d0 0x7fd5e40cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9c9a640 1 -- 172.21.15.6:0/3314411212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e40b40d0 msgr2=0x7fd5e40cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9c9a640 1 --2- 172.21.15.6:0/3314411212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e40b40d0 0x7fd5e40cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9c9a640 1 -- 172.21.15.6:0/3314411212 >> v1:172.21.15.6:6789/0 conn(0x7fd5e415a440 legacy=0x7fd5e40721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9499640 1 -- 172.21.15.6:0/3314411212 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223362315 0 0) 0x7fd5e406da60 con 0x7fd5e415a440 2024-03-18T14:43:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.333+0000 7fd5e9c9a640 1 -- 172.21.15.6:0/3314411212 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e406e300 con 0x7fd5e4072900 2024-03-18T14:43:53.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.334+0000 7fd5e9c9a640 1 --2- 172.21.15.6:0/3314411212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e40b3b90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000bb40 tx=0x7fd5e002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14dcc4ae00763b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.334+0000 7fd5e9499640 1 -- 172.21.15.6:0/3314411212 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e003b070 con 0x7fd5e4072900 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.334+0000 7fd5e9499640 1 -- 172.21.15.6:0/3314411212 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e0036440 con 0x7fd5e4072900 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.334+0000 7fd5e9499640 1 -- 172.21.15.6:0/3314411212 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0036810 con 0x7fd5e4072900 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3314411212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 msgr2=0x7fd5e40b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3314411212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e40b3b90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000bb40 tx=0x7fd5e002e150 comp rx=0 tx=0).stop 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3314411212 shutdown_connections 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3314411212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e40b40d0 0x7fd5e40cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3314411212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e40b3b90 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3314411212 >> 172.21.15.6:0/3314411212 conn(0x7fd5e40b1570 msgr2=0x7fd5e40b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3314411212 shutdown_connections 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.335+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3314411212 wait complete. 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.336+0000 7fd5ec6ff640 1 Processor -- start 2024-03-18T14:43:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.336+0000 7fd5ec6ff640 1 -- start start 2024-03-18T14:43:53.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ec6ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e4148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ec6ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5e40b40d0 0x7fd5e4145d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ec6ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e415a440 0x7fd5e4146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ec6ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e415a820 con 0x7fd5e4072900 2024-03-18T14:43:53.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ec6ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e40b4810 con 0x7fd5e40b40d0 2024-03-18T14:43:53.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ec6ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e4146fb0 con 0x7fd5e415a440 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e4148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e4148050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36296/0 (socket says 172.21.15.6:36296) 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 -- 172.21.15.6:0/3807850300 learned_addr learned my addr 172.21.15.6:0/3807850300 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5eac9c640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e415a440 0x7fd5e4146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5e9c9a640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5e40b40d0 0x7fd5e4145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 -- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e415a440 msgr2=0x7fd5e4146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e415a440 0x7fd5e4146660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 -- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5e40b40d0 msgr2=0x7fd5e4145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5e40b40d0 0x7fd5e4145d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 -- 172.21.15.6:0/3807850300 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e4147290 con 0x7fd5e4072900 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.337+0000 7fd5ea49b640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e4148050 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400c410 tx=0x7fd5d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.338+0000 7fd5daffd640 1 -- 172.21.15.6:0/3807850300 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40074e0 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.338+0000 7fd5daffd640 1 -- 172.21.15.6:0/3807850300 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d4007680 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.338+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e4147570 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.338+0000 7fd5daffd640 1 -- 172.21.15.6:0/3807850300 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4004a00 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.338+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e4147b30 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.339+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e404f5b0 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.339+0000 7fd5daffd640 1 -- 172.21.15.6:0/3807850300 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd5d401a020 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.339+0000 7fd5daffd640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd5c806bbd0 0x7fd5c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.340+0000 7fd5daffd640 1 -- 172.21.15.6:0/3807850300 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fd5d408a390 con 0x7fd5e4072900 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.340+0000 7fd5e9c9a640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd5c806bbd0 0x7fd5c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.340+0000 7fd5e9c9a640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd5c806bbd0 0x7fd5c806e090 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0038000 tx=0x7fd5e0032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.344+0000 7fd5daffd640 1 -- 172.21.15.6:0/3807850300 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd5d4010300 con 0x7fd5e4072900 2024-03-18T14:43:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.537+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd5e40afef0 con 0x7fd5e4072900 2024-03-18T14:43:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.538+0000 7fd5daffd640 1 -- 172.21.15.6:0/3807850300 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7fd5d4058d80 con 0x7fd5e4072900 2024-03-18T14:43:53.540 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd5c806bbd0 msgr2=0x7fd5c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd5c806bbd0 0x7fd5c806e090 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0038000 tx=0x7fd5e0032000 comp rx=0 tx=0).stop 2024-03-18T14:43:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 msgr2=0x7fd5e4148050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e4148050 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400c410 tx=0x7fd5d400c8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 shutdown_connections 2024-03-18T14:43:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd5c806bbd0 0x7fd5c806e090 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5e415a440 0x7fd5e4146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5e40b40d0 0x7fd5e4145d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 --2- 172.21.15.6:0/3807850300 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5e4072900 0x7fd5e4148050 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.539+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 >> 172.21.15.6:0/3807850300 conn(0x7fd5e40b1570 msgr2=0x7fd5e41594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.540+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 shutdown_connections 2024-03-18T14:43:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.540+0000 7fd5ec6ff640 1 -- 172.21.15.6:0/3807850300 wait complete. 2024-03-18T14:43:53.552 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:43:53.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:53.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:53.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[59919]: fuse finished with error 0 and tester_r 0 2024-03-18T14:43:53.581 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:43:53.666 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:43:53.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:53.666 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:43:53.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:43:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 Processor -- start 2024-03-18T14:43:53.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 -- start start 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faa3415bc80 0x7faa340cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa340ccbe0 0x7faa340ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa3415c060 0x7faa340ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7faa340c56e0 con 0x7faa3415c060 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7faa340cd0a0 con 0x7faa3415bc80 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3a2ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faa340cd380 con 0x7faa340ccbe0 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3889c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa3415c060 0x7faa340ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.820+0000 7faa3889c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa3415c060 0x7faa340ce9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55758/0 (socket says 172.21.15.6:55758) 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa3889c640 1 -- 172.21.15.6:0/3005794613 learned_addr learned my addr 172.21.15.6:0/3005794613 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:53.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa337fe640 1 --2- 172.21.15.6:0/3005794613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa340ccbe0 0x7faa340ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa33fff640 1 --2- 172.21.15.6:0/3005794613 >> v2:172.21.15.92:3300/0 conn(0x7faa3415bc80 0x7faa340cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa3889c640 1 -- 172.21.15.6:0/3005794613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa340ccbe0 msgr2=0x7faa340ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa3889c640 1 --2- 172.21.15.6:0/3005794613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa340ccbe0 0x7faa340ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa3889c640 1 -- 172.21.15.6:0/3005794613 >> v2:172.21.15.92:3300/0 conn(0x7faa3415bc80 msgr2=0x7faa340cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa3889c640 1 --2- 172.21.15.6:0/3005794613 >> v2:172.21.15.92:3300/0 conn(0x7faa3415bc80 0x7faa340cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa3889c640 1 -- 172.21.15.6:0/3005794613 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa340cd660 con 0x7faa3415c060 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.821+0000 7faa3889c640 1 --2- 172.21.15.6:0/3005794613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415c060 0x7faa340ce9a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faa340b4690 tx=0x7faa2c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f71ead2623c646 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa32ffd640 1 -- 172.21.15.6:0/3005794613 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c02c9e0 con 0x7faa3415c060 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa32ffd640 1 -- 172.21.15.6:0/3005794613 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa2c02cb80 con 0x7faa3415c060 2024-03-18T14:43:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa32ffd640 1 -- 172.21.15.6:0/3005794613 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c0343f0 con 0x7faa3415c060 2024-03-18T14:43:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3005794613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415c060 msgr2=0x7faa340ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3005794613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415c060 0x7faa340ce9a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faa340b4690 tx=0x7faa2c02be40 comp rx=0 tx=0).stop 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3005794613 shutdown_connections 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3005794613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415c060 0x7faa340ce9a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3005794613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa340ccbe0 0x7faa340ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3005794613 >> v2:172.21.15.92:3300/0 conn(0x7faa3415bc80 0x7faa340cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3005794613 >> 172.21.15.6:0/3005794613 conn(0x7faa340d1b90 msgr2=0x7faa340d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3005794613 shutdown_connections 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.822+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3005794613 wait complete. 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.824+0000 7faa3a2ff640 1 Processor -- start 2024-03-18T14:43:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.824+0000 7faa3a2ff640 1 -- start start 2024-03-18T14:43:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.825+0000 7faa3a2ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa340ccbe0 0x7faa341f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.825+0000 7faa3a2ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415bc80 0x7faa340b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.825+0000 7faa3a2ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa3415c060 0x7faa34152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa3a2ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faa340d0870 con 0x7faa3415bc80 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa3a2ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faa340d06f0 con 0x7faa340ccbe0 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa3a2ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faa340d09f0 con 0x7faa3415c060 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415bc80 0x7faa340b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415bc80 0x7faa340b6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55760/0 (socket says 172.21.15.6:55760) 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 -- 172.21.15.6:0/3117946527 learned_addr learned my addr 172.21.15.6:0/3117946527 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa3889c640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa3415c060 0x7faa34152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa33fff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa340ccbe0 0x7faa341f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 -- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa3415c060 msgr2=0x7faa34152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa3415c060 0x7faa34152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 -- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa340ccbe0 msgr2=0x7faa341f16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa340ccbe0 0x7faa341f16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 -- 172.21.15.6:0/3117946527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa340b4810 con 0x7faa3415bc80 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa337fe640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415bc80 0x7faa340b6d00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7faa2800d830 tx=0x7faa2800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa30ff9640 1 -- 172.21.15.6:0/3117946527 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2800b6c0 con 0x7faa3415bc80 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa30ff9640 1 -- 172.21.15.6:0/3117946527 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa28019070 con 0x7faa3415bc80 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa340b4b50 con 0x7faa3415bc80 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa30ff9640 1 -- 172.21.15.6:0/3117946527 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2800ba80 con 0x7faa3415bc80 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.826+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa340b5390 con 0x7faa3415bc80 2024-03-18T14:43:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.827+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa3409fc00 con 0x7faa3415bc80 2024-03-18T14:43:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.828+0000 7faa30ff9640 1 -- 172.21.15.6:0/3117946527 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faa2801e460 con 0x7faa3415bc80 2024-03-18T14:43:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.828+0000 7faa30ff9640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2006bbd0 0x7faa2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.828+0000 7faa30ff9640 1 -- 172.21.15.6:0/3117946527 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7faa2808b230 con 0x7faa3415bc80 2024-03-18T14:43:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.828+0000 7faa33fff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2006bbd0 0x7faa2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:53.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.829+0000 7faa33fff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2006bbd0 0x7faa2006e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faa24006fd0 tx=0x7faa24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:53.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:53.833+0000 7faa30ff9640 1 -- 172.21.15.6:0/3117946527 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faa340b4b50 con 0x7faa3415bc80 2024-03-18T14:43:54.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.054+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faa3404f620 con 0x7faa3415bc80 2024-03-18T14:43:54.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.054+0000 7faa30ff9640 1 -- 172.21.15.6:0/3117946527 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7faa280594a0 con 0x7faa3415bc80 2024-03-18T14:43:54.056 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2006bbd0 msgr2=0x7faa2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2006bbd0 0x7faa2006e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faa24006fd0 tx=0x7faa24008040 comp rx=0 tx=0).stop 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415bc80 msgr2=0x7faa340b6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415bc80 0x7faa340b6d00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7faa2800d830 tx=0x7faa2800dcf0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 shutdown_connections 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faa2006bbd0 0x7faa2006e090 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa3415c060 0x7faa34152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa3415bc80 0x7faa340b6d00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 --2- 172.21.15.6:0/3117946527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa340ccbe0 0x7faa341f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:54.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 >> 172.21.15.6:0/3117946527 conn(0x7faa340d1b90 msgr2=0x7faa340c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:54.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 shutdown_connections 2024-03-18T14:43:54.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54.056+0000 7faa3a2ff640 1 -- 172.21.15.6:0/3117946527 wait complete. 2024-03-18T14:43:54.069 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:43:54.069 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:43:54.069 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:43:54.069 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:43:54.069 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:54.070 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:43:54.070 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:43:54.070 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:43:54.070 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:54.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.070 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:54.085 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42531sec preferred_lft 42531sec 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:54.086 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c41d:adff:fe29:e32/64 scope link 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::d035:11ff:fe58:6e4e/64 scope link 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:54.087 INFO:teuthology.orchestra.run.smithi006.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:43:54.088 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:54.088 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::9023:d7ff:fecf:2a25/64 scope link 2024-03-18T14:43:54.088 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:43:54.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.088 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:43:54.150 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:43:54.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.152 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:43:54.206 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:43:54.207 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:43:54.207 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:43:54.207 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:43:54.207 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:43:54.207 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:43:54.207 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:43:54.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:43:54.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:43:54.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:43:54.207 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:43:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:43:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:54.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.345 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:43:54.359 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T14:43:54.359 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:43:54.359 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:54.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:43:54.425 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T14:43:54.425 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:54.425 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:54.479 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:43:54.479 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:43:54.479 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:54.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:54.603 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:54.658 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:54.658 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:54.658 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:54.658 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:54.658 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:54.658 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:54.658 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:54.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.661 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:54.712 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:43:54.712 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T14:43:54.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:43:54.784 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:43:54.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:2024-03-18T14:43:54.838+0000 7feb42446480 -1 init, newargv = 0x56141a6d42a0 newargc=15 2024-03-18T14:43:54.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[60883]: starting ceph client 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:43:54.841 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:43:54.842 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:54.843 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:43:54.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.844 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:43:54.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[60883]: starting fuse 2024-03-18T14:43:54.859 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:43:54.859 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:43:54.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:43:54.859 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:43:54.859 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:54.860 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.0")) 2024-03-18T14:43:54.861 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:43:54.979 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.0.60883.asok 2024-03-18T14:43:54.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:54.990 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60883.asok 2024-03-18T14:43:54.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:54.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60883.asok status 2024-03-18T14:43:55.129 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:43:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:43:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:43:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "0", 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "60883", 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 6477, 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6477 2024-03-18T14:43:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2011493348 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2011493348 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.6477 192.168.0.1:0/2011493348", 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/2011493348", 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:43:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 25, 2024-03-18T14:43:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 56, 2024-03-18T14:43:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:43:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:43:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:43:55.132 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:43:55.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:55.138 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:43:55.138 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:55.154 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:43:55.154 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:55.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:55.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> import errno 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> ret = os.read(fd, 1024) 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> else: 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> os.write(fd, b'"'"'content'"'"') 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> except OSError as e: 2024-03-18T14:43:55.221 DEBUG:teuthology.orchestra.run.smithi006:> if e.errno != errno.EPERM: 2024-03-18T14:43:55.222 DEBUG:teuthology.orchestra.run.smithi006:> raise 2024-03-18T14:43:55.222 DEBUG:teuthology.orchestra.run.smithi006:> else: 2024-03-18T14:43:55.222 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("client does not check permission of data pool") 2024-03-18T14:43:55.222 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:43:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:43:55.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2024-03-18T14:43:55.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.489+0000 7f00235b2640 1 Processor -- start 2024-03-18T14:43:55.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.489+0000 7f00235b2640 1 -- start start 2024-03-18T14:43:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f00235b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c06d1f0 0x7f001c0d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f00235b2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f001c0d0680 0x7f001c15d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f00235b2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f001c0d0f30 con 0x7f001c0ce930 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f00235b2640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f001c0b1070 con 0x7f001c0d0680 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f00235b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f001c0b1350 con 0x7f001c06d1f0 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0021b4f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f001c0d0680 0x7f001c15d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0021b4f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f001c0d0680 0x7f001c15d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55766/0 (socket says 172.21.15.6:55766) 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0021b4f640 1 -- 172.21.15.6:0/4041463122 learned_addr learned my addr 172.21.15.6:0/4041463122 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0020b4d640 1 --2- 172.21.15.6:0/4041463122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c06d1f0 0x7f001c0d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0021b4f640 1 -- 172.21.15.6:0/4041463122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c06d1f0 msgr2=0x7f001c0d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0021b4f640 1 --2- 172.21.15.6:0/4041463122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c06d1f0 0x7f001c0d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0021b4f640 1 -- 172.21.15.6:0/4041463122 >> v1:172.21.15.92:6789/0 conn(0x7f001c0ce930 legacy=0x7f001c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0021b4f640 1 -- 172.21.15.6:0/4041463122 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c0d0b10 con 0x7f001c0d0680 2024-03-18T14:43:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.490+0000 7f0020b4d640 1 --2- 172.21.15.6:0/4041463122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c06d1f0 0x7f001c0d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.491+0000 7f0021b4f640 1 --2- 172.21.15.6:0/4041463122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15d6b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f001402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba18fdc8954cc9a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.491+0000 7f0013fff640 1 -- 172.21.15.6:0/4041463122 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001402ca70 con 0x7f001c0d0680 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.491+0000 7f0013fff640 1 -- 172.21.15.6:0/4041463122 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001402cc10 con 0x7f001c0d0680 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.491+0000 7f0013fff640 1 -- 172.21.15.6:0/4041463122 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00140343f0 con 0x7f001c0d0680 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 -- 172.21.15.6:0/4041463122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 msgr2=0x7f001c15d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 --2- 172.21.15.6:0/4041463122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15d6b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f001402bed0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 -- 172.21.15.6:0/4041463122 shutdown_connections 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 --2- 172.21.15.6:0/4041463122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15d6b0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 --2- 172.21.15.6:0/4041463122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c06d1f0 0x7f001c0d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 -- 172.21.15.6:0/4041463122 >> 172.21.15.6:0/4041463122 conn(0x7f001c08b0c0 msgr2=0x7f001c0d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:55.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 -- 172.21.15.6:0/4041463122 shutdown_connections 2024-03-18T14:43:55.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.492+0000 7f00235b2640 1 -- 172.21.15.6:0/4041463122 wait complete. 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.493+0000 7f00235b2640 1 Processor -- start 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.493+0000 7f00235b2640 1 -- start start 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f00235b2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f001c06d1f0 0x7f001c15aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f00235b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c0ce930 0x7f001c15e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f00235b2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f00235b2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f001c15dbf0 con 0x7f001c0d0680 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f00235b2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f001c0cef20 con 0x7f001c06d1f0 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f00235b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f001c15dd70 con 0x7f001c0ce930 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55780/0 (socket says 172.21.15.6:55780) 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 -- 172.21.15.6:0/1330563826 learned_addr learned my addr 172.21.15.6:0/1330563826 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0020b4d640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c0ce930 0x7f001c15e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 -- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c0ce930 msgr2=0x7f001c15e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c0ce930 0x7f001c15e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f002134e640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f001c06d1f0 0x7f001c15aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 -- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f001c06d1f0 msgr2=0x7f001c15aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f001c06d1f0 0x7f001c15aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 -- 172.21.15.6:0/1330563826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c1f4480 con 0x7f001c0d0680 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.494+0000 7f0021b4f640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15e6c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f0014004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.495+0000 7f0011ffb640 1 -- 172.21.15.6:0/1330563826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014043020 con 0x7f001c0d0680 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.495+0000 7f0011ffb640 1 -- 172.21.15.6:0/1330563826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001403e450 con 0x7f001c0d0680 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.495+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001c1f4760 con 0x7f001c0d0680 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.495+0000 7f0011ffb640 1 -- 172.21.15.6:0/1330563826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001403e820 con 0x7f001c0d0680 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.495+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f001c0aeb00 con 0x7f001c0d0680 2024-03-18T14:43:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.496+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f001c0751b0 con 0x7f001c0d0680 2024-03-18T14:43:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.496+0000 7f0011ffb640 1 -- 172.21.15.6:0/1330563826 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0014046020 con 0x7f001c0d0680 2024-03-18T14:43:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.497+0000 7f0011ffb640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000006bbd0 0x7f000006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.497+0000 7f0011ffb640 1 -- 172.21.15.6:0/1330563826 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f00140ad600 con 0x7f001c0d0680 2024-03-18T14:43:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.497+0000 7f002134e640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000006bbd0 0x7f000006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.497+0000 7f002134e640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000006bbd0 0x7f000006e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f001c15d320 tx=0x7f0018009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.501+0000 7f0011ffb640 1 -- 172.21.15.6:0/1330563826 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f001404d030 con 0x7f001c0d0680 2024-03-18T14:43:55.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.753+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) -- 0x7f001c08b680 con 0x7f001c0d0680 2024-03-18T14:43:55.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.755+0000 7f0011ffb640 1 -- 172.21.15.6:0/1330563826 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v17) ==== 183+0+140 (secure 0 0 0) 0x7f001407bec0 con 0x7f001c0d0680 2024-03-18T14:43:55.758 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:43:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:43:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-18T14:43:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:43:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow w pool=cephfs_data" 2024-03-18T14:43:55.758 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:43:55.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000006bbd0 msgr2=0x7f000006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000006bbd0 0x7f000006e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f001c15d320 tx=0x7f0018009040 comp rx=0 tx=0).stop 2024-03-18T14:43:55.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 msgr2=0x7f001c15e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15e6c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f0014004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 shutdown_connections 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f000006bbd0 0x7f000006e090 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f001c0d0680 0x7f001c15e6c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f001c0ce930 0x7f001c15e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 --2- 172.21.15.6:0/1330563826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f001c06d1f0 0x7f001c15aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 >> 172.21.15.6:0/1330563826 conn(0x7f001c08b0c0 msgr2=0x7f001c0709b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 shutdown_connections 2024-03-18T14:43:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.758+0000 7f00235b2640 1 -- 172.21.15.6:0/1330563826 wait complete. 2024-03-18T14:43:55.773 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:55.773 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:55.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.926+0000 7f9ee0b13640 1 Processor -- start 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.926+0000 7f9ee0b13640 1 -- start start 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9ee0b13640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9edc15bd40 0x7f9edc0d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9ee0b13640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9edc0cfea0 0x7f9edc0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9ee0b13640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9edc0cc870 con 0x7f9edc0d1d90 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9ee0b13640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9edc0d02d0 con 0x7f9edc0cfea0 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9ee0b13640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f9edc0d0500 con 0x7f9edc15bd40 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9edad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9edc0cfea0 0x7f9edc0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9ed3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9edc0d1d90 0x7f9edc0cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35814/0 (socket says 172.21.15.6:35814) 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9ed3fff640 1 -- 172.21.15.6:0/4134788544 learned_addr learned my addr 172.21.15.6:0/4134788544 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.927+0000 7f9eda59c640 1 --2- 172.21.15.6:0/4134788544 >> v2:172.21.15.92:3300/0 conn(0x7f9edc15bd40 0x7f9edc0d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.928+0000 7f9edad9d640 1 -- 172.21.15.6:0/4134788544 >> v2:172.21.15.92:3300/0 conn(0x7f9edc15bd40 msgr2=0x7f9edc0d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.928+0000 7f9edad9d640 1 --2- 172.21.15.6:0/4134788544 >> v2:172.21.15.92:3300/0 conn(0x7f9edc15bd40 0x7f9edc0d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.928+0000 7f9edad9d640 1 -- 172.21.15.6:0/4134788544 >> v1:172.21.15.6:6789/0 conn(0x7f9edc0d1d90 legacy=0x7f9edc0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.928+0000 7f9edad9d640 1 -- 172.21.15.6:0/4134788544 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc0d07e0 con 0x7f9edc0cfea0 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.928+0000 7f9ed9d9b640 1 -- 172.21.15.6:0/4134788544 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171917007 0 0) 0x7f9edc0cc870 con 0x7f9edc0d1d90 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.928+0000 7f9edad9d640 1 --2- 172.21.15.6:0/4134788544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc0cfea0 0x7f9edc0cd030 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0cca40 tx=0x7f9ed402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c89e2f00f27bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ed9d9b640 1 -- 172.21.15.6:0/4134788544 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4039070 con 0x7f9edc0cfea0 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ed9d9b640 1 -- 172.21.15.6:0/4134788544 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed402cc60 con 0x7f9edc0cfea0 2024-03-18T14:43:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ed9d9b640 1 -- 172.21.15.6:0/4134788544 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4034660 con 0x7f9edc0cfea0 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 -- 172.21.15.6:0/4134788544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc0cfea0 msgr2=0x7f9edc0cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/4134788544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc0cfea0 0x7f9edc0cd030 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0cca40 tx=0x7f9ed402beb0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 -- 172.21.15.6:0/4134788544 shutdown_connections 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/4134788544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc0cfea0 0x7f9edc0cd030 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/4134788544 >> v2:172.21.15.92:3300/0 conn(0x7f9edc15bd40 0x7f9edc0d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 -- 172.21.15.6:0/4134788544 >> 172.21.15.6:0/4134788544 conn(0x7f9edc09ae20 msgr2=0x7f9edc08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 -- 172.21.15.6:0/4134788544 shutdown_connections 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.929+0000 7f9ee0b13640 1 -- 172.21.15.6:0/4134788544 wait complete. 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 Processor -- start 2024-03-18T14:43:55.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 -- start start 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9edc0cfea0 0x7f9edc1f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9edc0d1d90 0x7f9edc157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc15bd40 0x7f9edc1579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc0d0ac0 con 0x7f9edc15bd40 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc0d2210 con 0x7f9edc0cfea0 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9ee0b13640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9edc0d0c40 con 0x7f9edc0d1d90 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9edad9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc15bd40 0x7f9edc1579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9edad9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc15bd40 0x7f9edc1579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55804/0 (socket says 172.21.15.6:55804) 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.931+0000 7f9edad9d640 1 -- 172.21.15.6:0/3572065222 learned_addr learned my addr 172.21.15.6:0/3572065222 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9eda59c640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9edc0cfea0 0x7f9edc1f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9ed3fff640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9edc0d1d90 0x7f9edc157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9edad9d640 1 -- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9edc0d1d90 msgr2=0x7f9edc157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9edad9d640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9edc0d1d90 0x7f9edc157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9edad9d640 1 -- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9edc0cfea0 msgr2=0x7f9edc1f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9edad9d640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9edc0cfea0 0x7f9edc1f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9edad9d640 1 -- 172.21.15.6:0/3572065222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc0b0df0 con 0x7f9edc15bd40 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9ed3fff640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9edc0d1d90 0x7f9edc157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9edad9d640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc15bd40 0x7f9edc1579c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009e50 tx=0x7f9ed4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9ed2ffd640 1 -- 172.21.15.6:0/3572065222 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4039050 con 0x7f9edc15bd40 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9ed2ffd640 1 -- 172.21.15.6:0/3572065222 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed4045070 con 0x7f9edc15bd40 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9ed2ffd640 1 -- 172.21.15.6:0/3572065222 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed403e830 con 0x7f9edc15bd40 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9edc0b10d0 con 0x7f9edc15bd40 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.932+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9edc1f13e0 con 0x7f9edc15bd40 2024-03-18T14:43:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.933+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9edc09fa90 con 0x7f9edc15bd40 2024-03-18T14:43:55.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.934+0000 7f9ed2ffd640 1 -- 172.21.15.6:0/3572065222 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9ed403e9d0 con 0x7f9edc15bd40 2024-03-18T14:43:55.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.934+0000 7f9ed2ffd640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:55.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.934+0000 7f9ed2ffd640 1 -- 172.21.15.6:0/3572065222 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f9ed40ada40 con 0x7f9edc15bd40 2024-03-18T14:43:55.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.934+0000 7f9eda59c640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:55.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.935+0000 7f9eda59c640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc006fd0 tx=0x7f9ecc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:55.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:55.939+0000 7f9ed2ffd640 1 -- 172.21.15.6:0/3572065222 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9ed407c550 con 0x7f9edc15bd40 2024-03-18T14:43:56.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.236+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9edc04f620 con 0x7f9edc15bd40 2024-03-18T14:43:56.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.237+0000 7f9ed2ffd640 1 -- 172.21.15.6:0/3572065222 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7f9ed407be50 con 0x7f9edc15bd40 2024-03-18T14:43:56.239 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:56.240 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:56.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ebc06bbd0 msgr2=0x7f9ebc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc006fd0 tx=0x7f9ecc008040 comp rx=0 tx=0).stop 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc15bd40 msgr2=0x7f9edc1579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc15bd40 0x7f9edc1579c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009e50 tx=0x7f9ed4004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 shutdown_connections 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9edc15bd40 0x7f9edc1579c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9edc0d1d90 0x7f9edc157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 --2- 172.21.15.6:0/3572065222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9edc0cfea0 0x7f9edc1f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 >> 172.21.15.6:0/3572065222 conn(0x7f9edc09ae20 msgr2=0x7f9edc073c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 shutdown_connections 2024-03-18T14:43:56.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.240+0000 7f9ee0b13640 1 -- 172.21.15.6:0/3572065222 wait complete. 2024-03-18T14:43:56.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:43:39.597879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:43:40.299491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:56.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:56.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.358+0000 7fc01d219640 1 Processor -- start 2024-03-18T14:43:56.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.358+0000 7fc01d219640 1 -- start start 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01d219640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc0180d0730 0x7fc0180cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01d219640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0180d2510 0x7fc01815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01d219640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0180cfac0 con 0x7fc0180ce9d0 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01d219640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc0180cd8d0 con 0x7fc0180d2510 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01d219640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc0180cccc0 con 0x7fc0180d0730 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01759e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0180d2510 0x7fc01815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01759e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0180d2510 0x7fc01815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55806/0 (socket says 172.21.15.6:55806) 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.359+0000 7fc01759e640 1 -- 172.21.15.6:0/401558885 learned_addr learned my addr 172.21.15.6:0/401558885 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.360+0000 7fc00ffff640 1 --2- 172.21.15.6:0/401558885 >> v2:172.21.15.92:3300/0 conn(0x7fc0180d0730 0x7fc0180cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.360+0000 7fc01759e640 1 -- 172.21.15.6:0/401558885 >> v2:172.21.15.92:3300/0 conn(0x7fc0180d0730 msgr2=0x7fc0180cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.360+0000 7fc01759e640 1 --2- 172.21.15.6:0/401558885 >> v2:172.21.15.92:3300/0 conn(0x7fc0180d0730 0x7fc0180cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.360+0000 7fc01759e640 1 -- 172.21.15.6:0/401558885 >> v1:172.21.15.92:6789/0 conn(0x7fc0180ce9d0 legacy=0x7fc0180d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:56.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.360+0000 7fc01759e640 1 -- 172.21.15.6:0/401558885 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0180ccfa0 con 0x7fc0180d2510 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.361+0000 7fc01759e640 1 --2- 172.21.15.6:0/401558885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d2510 0x7fc01815b690 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc0180d19a0 tx=0x7fc01002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1e6a892f4f9a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.361+0000 7fc01659c640 1 -- 172.21.15.6:0/401558885 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01002c9f0 con 0x7fc0180d2510 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.361+0000 7fc01659c640 1 -- 172.21.15.6:0/401558885 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc01002cb90 con 0x7fc0180d2510 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.361+0000 7fc01659c640 1 -- 172.21.15.6:0/401558885 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0100343f0 con 0x7fc0180d2510 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.361+0000 7fc01d219640 1 -- 172.21.15.6:0/401558885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d2510 msgr2=0x7fc01815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.361+0000 7fc01d219640 1 --2- 172.21.15.6:0/401558885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d2510 0x7fc01815b690 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc0180d19a0 tx=0x7fc01002be50 comp rx=0 tx=0).stop 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.362+0000 7fc01d219640 1 -- 172.21.15.6:0/401558885 shutdown_connections 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.362+0000 7fc01d219640 1 --2- 172.21.15.6:0/401558885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d2510 0x7fc01815b690 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.362+0000 7fc01d219640 1 --2- 172.21.15.6:0/401558885 >> v2:172.21.15.92:3300/0 conn(0x7fc0180d0730 0x7fc0180cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.362+0000 7fc01d219640 1 -- 172.21.15.6:0/401558885 >> 172.21.15.6:0/401558885 conn(0x7fc01808b0c0 msgr2=0x7fc0180d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.362+0000 7fc01d219640 1 -- 172.21.15.6:0/401558885 shutdown_connections 2024-03-18T14:43:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.362+0000 7fc01d219640 1 -- 172.21.15.6:0/401558885 wait complete. 2024-03-18T14:43:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.363+0000 7fc01d219640 1 Processor -- start 2024-03-18T14:43:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.363+0000 7fc01d219640 1 -- start start 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01d219640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0180ce9d0 0x7fc01815a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01d219640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d0730 0x7fc0180af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01d219640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0180d2510 0x7fc0180c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01d219640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc0180d28f0 con 0x7fc0180d0730 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01d219640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc0180cd560 con 0x7fc0180d2510 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01d219640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc0180cedb0 con 0x7fc0180ce9d0 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d0730 0x7fc0180af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d0730 0x7fc0180af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55822/0 (socket says 172.21.15.6:55822) 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 -- 172.21.15.6:0/194111691 learned_addr learned my addr 172.21.15.6:0/194111691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc016d9d640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0180ce9d0 0x7fc01815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01759e640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0180d2510 0x7fc0180c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 -- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0180ce9d0 msgr2=0x7fc01815a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0180ce9d0 0x7fc01815a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 -- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0180d2510 msgr2=0x7fc0180c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0180d2510 0x7fc0180c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 -- 172.21.15.6:0/194111691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0181592a0 con 0x7fc0180d0730 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc016d9d640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0180ce9d0 0x7fc01815a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01759e640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0180d2510 0x7fc0180c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00ffff640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d0730 0x7fc0180af8d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc00000c7f0 tx=0x7fc00000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00f7fe640 1 -- 172.21.15.6:0/194111691 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000015070 con 0x7fc0180d0730 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc00f7fe640 1 -- 172.21.15.6:0/194111691 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc000004240 con 0x7fc0180d0730 2024-03-18T14:43:56.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.364+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc018159580 con 0x7fc0180d0730 2024-03-18T14:43:56.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.365+0000 7fc00f7fe640 1 -- 172.21.15.6:0/194111691 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000007620 con 0x7fc0180d0730 2024-03-18T14:43:56.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.365+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc018159b40 con 0x7fc0180d0730 2024-03-18T14:43:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.366+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0180751b0 con 0x7fc0180d0730 2024-03-18T14:43:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.366+0000 7fc00f7fe640 1 -- 172.21.15.6:0/194111691 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc0000077c0 con 0x7fc0180d0730 2024-03-18T14:43:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.367+0000 7fc00f7fe640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbff806bbd0 0x7fbff806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.367+0000 7fc00f7fe640 1 -- 172.21.15.6:0/194111691 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fc00008ad70 con 0x7fc0180d0730 2024-03-18T14:43:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.367+0000 7fc016d9d640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbff806bbd0 0x7fbff806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.367+0000 7fc016d9d640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbff806bbd0 0x7fbff806e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc008001760 tx=0x7fc008009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.371+0000 7fc00f7fe640 1 -- 172.21.15.6:0/194111691 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc000059760 con 0x7fc0180d0730 2024-03-18T14:43:56.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.551+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0180cfe00 con 0x7fc0180d0730 2024-03-18T14:43:56.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.551+0000 7fc00f7fe640 1 -- 172.21.15.6:0/194111691 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fc000059060 con 0x7fc0180d0730 2024-03-18T14:43:56.553 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbff806bbd0 msgr2=0x7fbff806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbff806bbd0 0x7fbff806e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc008001760 tx=0x7fc008009040 comp rx=0 tx=0).stop 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d0730 msgr2=0x7fc0180af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d0730 0x7fc0180af8d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc00000c7f0 tx=0x7fc00000ccb0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 shutdown_connections 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fbff806bbd0 0x7fbff806e090 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0180d2510 0x7fc0180c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0180d0730 0x7fc0180af8d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 --2- 172.21.15.6:0/194111691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0180ce9d0 0x7fc01815a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 >> 172.21.15.6:0/194111691 conn(0x7fc01808b0c0 msgr2=0x7fc01806c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 shutdown_connections 2024-03-18T14:43:56.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.553+0000 7fc01d219640 1 -- 172.21.15.6:0/194111691 wait complete. 2024-03-18T14:43:56.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.669+0000 7f1fc5097640 1 Processor -- start 2024-03-18T14:43:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.669+0000 7f1fc5097640 1 -- start start 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fc5097640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1fc00d0730 0x7f1fc00cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fc5097640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1fc00d2510 0x7f1fc015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fc5097640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc00cfac0 con 0x7f1fc00ce9d0 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fc5097640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1fc00cd8d0 con 0x7f1fc00d2510 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fc5097640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1fc00cccc0 con 0x7f1fc00d0730 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fbed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1fc00ce9d0 0x7f1fc00d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35824/0 (socket says 172.21.15.6:35824) 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fbf59e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1fc00d2510 0x7f1fc015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fbed9d640 1 -- 172.21.15.6:0/2857202014 learned_addr learned my addr 172.21.15.6:0/2857202014 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fb7fff640 1 --2- 172.21.15.6:0/2857202014 >> v2:172.21.15.92:3300/0 conn(0x7f1fc00d0730 0x7f1fc00cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fbe59c640 1 -- 172.21.15.6:0/2857202014 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699922933 0 0) 0x7f1fc00cfac0 con 0x7f1fc00ce9d0 2024-03-18T14:43:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fbe59c640 1 -- 172.21.15.6:0/2857202014 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fac003520 con 0x7f1fc00ce9d0 2024-03-18T14:43:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fbf59e640 1 -- 172.21.15.6:0/2857202014 >> v2:172.21.15.92:3300/0 conn(0x7f1fc00d0730 msgr2=0x7f1fc00cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.670+0000 7f1fbf59e640 1 --2- 172.21.15.6:0/2857202014 >> v2:172.21.15.92:3300/0 conn(0x7f1fc00d0730 0x7f1fc00cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.671+0000 7f1fbf59e640 1 -- 172.21.15.6:0/2857202014 >> v1:172.21.15.6:6789/0 conn(0x7f1fc00ce9d0 legacy=0x7f1fc00d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.671+0000 7f1fbf59e640 1 -- 172.21.15.6:0/2857202014 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc00ccfa0 con 0x7f1fc00d2510 2024-03-18T14:43:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.671+0000 7f1fbf59e640 1 --2- 172.21.15.6:0/2857202014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d2510 0x7f1fc015b690 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00d19a0 tx=0x7f1fb802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d776c5b20f98b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.671+0000 7f1fbe59c640 1 -- 172.21.15.6:0/2857202014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8039070 con 0x7f1fc00d2510 2024-03-18T14:43:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.671+0000 7f1fbe59c640 1 -- 172.21.15.6:0/2857202014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fb802cbf0 con 0x7f1fc00d2510 2024-03-18T14:43:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.671+0000 7f1fbe59c640 1 -- 172.21.15.6:0/2857202014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8034440 con 0x7f1fc00d2510 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 -- 172.21.15.6:0/2857202014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d2510 msgr2=0x7f1fc015b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 --2- 172.21.15.6:0/2857202014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d2510 0x7f1fc015b690 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00d19a0 tx=0x7f1fb802be40 comp rx=0 tx=0).stop 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 -- 172.21.15.6:0/2857202014 shutdown_connections 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 --2- 172.21.15.6:0/2857202014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d2510 0x7f1fc015b690 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 --2- 172.21.15.6:0/2857202014 >> v2:172.21.15.92:3300/0 conn(0x7f1fc00d0730 0x7f1fc00cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 -- 172.21.15.6:0/2857202014 >> 172.21.15.6:0/2857202014 conn(0x7f1fc008b0c0 msgr2=0x7f1fc00d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 -- 172.21.15.6:0/2857202014 shutdown_connections 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.672+0000 7f1fc5097640 1 -- 172.21.15.6:0/2857202014 wait complete. 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.673+0000 7f1fc5097640 1 Processor -- start 2024-03-18T14:43:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.673+0000 7f1fc5097640 1 -- start start 2024-03-18T14:43:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fc5097640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fc00ce9d0 0x7f1fc006dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fc5097640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d0730 0x7f1fc0155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fc5097640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fc00d2510 0x7f1fc00c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fc5097640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc00d28f0 con 0x7f1fc00d0730 2024-03-18T14:43:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fc5097640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc00cd560 con 0x7f1fc00ce9d0 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fc5097640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc00cedb0 con 0x7f1fc00d2510 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fb7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d0730 0x7f1fc0155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fb7fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d0730 0x7f1fc0155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55842/0 (socket says 172.21.15.6:55842) 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fb7fff640 1 -- 172.21.15.6:0/1833000125 learned_addr learned my addr 172.21.15.6:0/1833000125 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.674+0000 7f1fbf59e640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fc00d2510 0x7f1fc00c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fbed9d640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fc00ce9d0 0x7f1fc006dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb7fff640 1 -- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fc00d2510 msgr2=0x7f1fc00c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb7fff640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fc00d2510 0x7f1fc00c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb7fff640 1 -- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fc00ce9d0 msgr2=0x7f1fc006dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb7fff640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fc00ce9d0 0x7f1fc006dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb7fff640 1 -- 172.21.15.6:0/1833000125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc006e110 con 0x7f1fc00d0730 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fbf59e640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fc00d2510 0x7f1fc00c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fbed9d640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fc00ce9d0 0x7f1fc006dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb7fff640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d0730 0x7f1fc0155180 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800d800 tx=0x7f1fa800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb77fe640 1 -- 172.21.15.6:0/1833000125 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800b6c0 con 0x7f1fc00d0730 2024-03-18T14:43:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb77fe640 1 -- 172.21.15.6:0/1833000125 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fa8019070 con 0x7f1fc00d0730 2024-03-18T14:43:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fb77fe640 1 -- 172.21.15.6:0/1833000125 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800ba80 con 0x7f1fc00d0730 2024-03-18T14:43:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc01f5050 con 0x7f1fc00d0730 2024-03-18T14:43:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.675+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc01f58f0 con 0x7f1fc00d0730 2024-03-18T14:43:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.676+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc00751b0 con 0x7f1fc00d0730 2024-03-18T14:43:56.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.677+0000 7f1fb77fe640 1 -- 172.21.15.6:0/1833000125 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1fa800bc20 con 0x7f1fc00d0730 2024-03-18T14:43:56.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.677+0000 7f1fb77fe640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1fa006bc00 0x7f1fa006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:56.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.677+0000 7f1fb77fe640 1 -- 172.21.15.6:0/1833000125 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f1fa808b140 con 0x7f1fc00d0730 2024-03-18T14:43:56.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.678+0000 7f1fbed9d640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1fa006bc00 0x7f1fa006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:56.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.678+0000 7f1fbed9d640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1fa006bc00 0x7f1fa006e0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0007fd0 tx=0x7f1fb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:56.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.682+0000 7f1fb77fe640 1 -- 172.21.15.6:0/1833000125 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1fa8059a00 con 0x7f1fc00d0730 2024-03-18T14:43:56.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.947+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1fc00cfe00 con 0x7f1fc00d0730 2024-03-18T14:43:56.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.948+0000 7f1fb77fe640 1 -- 172.21.15.6:0/1833000125 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f1fa8059300 con 0x7f1fc00d0730 2024-03-18T14:43:56.950 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:56.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1fa006bc00 msgr2=0x7f1fa006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1fa006bc00 0x7f1fa006e0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0007fd0 tx=0x7f1fb0009040 comp rx=0 tx=0).stop 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d0730 msgr2=0x7f1fc0155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d0730 0x7f1fc0155180 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800d800 tx=0x7f1fa800dcc0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 shutdown_connections 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1fa006bc00 0x7f1fa006e0c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fc00d2510 0x7f1fc00c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fc00d0730 0x7f1fc0155180 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 --2- 172.21.15.6:0/1833000125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fc00ce9d0 0x7f1fc006dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 >> 172.21.15.6:0/1833000125 conn(0x7f1fc008b0c0 msgr2=0x7f1fc00ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 shutdown_connections 2024-03-18T14:43:56.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:56.950+0000 7f1fc5097640 1 -- 172.21.15.6:0/1833000125 wait complete. 2024-03-18T14:43:56.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.066+0000 7fda8fb62640 1 Processor -- start 2024-03-18T14:43:57.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8fb62640 1 -- start start 2024-03-18T14:43:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8fb62640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda8806b430 0x7fda880ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8fb62640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda880b22c0 con 0x7fda8806b810 2024-03-18T14:43:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8fb62640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8815e650 con 0x7fda8815d640 2024-03-18T14:43:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8fb62640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fda8815e930 con 0x7fda8806b430 2024-03-18T14:43:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8d8fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda8806b430 0x7fda880ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8d0fd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fda8806b810 0x7fda880b1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35838/0 (socket says 172.21.15.6:35838) 2024-03-18T14:43:57.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.067+0000 7fda8d0fd640 1 -- 172.21.15.6:0/3483606374 learned_addr learned my addr 172.21.15.6:0/3483606374 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:57.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.068+0000 7fda8d8fe640 1 -- 172.21.15.6:0/3483606374 >> v1:172.21.15.92:6789/0 conn(0x7fda8815d640 legacy=0x7fda8815daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:57.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.068+0000 7fda8d8fe640 1 -- 172.21.15.6:0/3483606374 >> v1:172.21.15.6:6789/0 conn(0x7fda8806b810 legacy=0x7fda880b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.068+0000 7fda8d8fe640 1 -- 172.21.15.6:0/3483606374 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8815ec10 con 0x7fda8806b430 2024-03-18T14:43:57.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.068+0000 7fda8c8fc640 1 -- 172.21.15.6:0/3483606374 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1570837873 0 0) 0x7fda880b22c0 con 0x7fda8806b810 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8d8fe640 1 --2- 172.21.15.6:0/3483606374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda880ce670 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fda840097a0 tx=0x7fda8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc302364080a3bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8c8fc640 1 -- 172.21.15.6:0/3483606374 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84035020 con 0x7fda8806b430 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8c8fc640 1 -- 172.21.15.6:0/3483606374 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda84031830 con 0x7fda8806b430 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8c8fc640 1 -- 172.21.15.6:0/3483606374 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84031c00 con 0x7fda8806b430 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8fb62640 1 -- 172.21.15.6:0/3483606374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 msgr2=0x7fda880ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8fb62640 1 --2- 172.21.15.6:0/3483606374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda880ce670 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fda840097a0 tx=0x7fda8402b8d0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8fb62640 1 -- 172.21.15.6:0/3483606374 shutdown_connections 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8fb62640 1 --2- 172.21.15.6:0/3483606374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda880ce670 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8fb62640 1 -- 172.21.15.6:0/3483606374 >> 172.21.15.6:0/3483606374 conn(0x7fda88099a50 msgr2=0x7fda880d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8fb62640 1 -- 172.21.15.6:0/3483606374 shutdown_connections 2024-03-18T14:43:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.069+0000 7fda8fb62640 1 -- 172.21.15.6:0/3483606374 wait complete. 2024-03-18T14:43:57.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 Processor -- start 2024-03-18T14:43:57.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 -- start start 2024-03-18T14:43:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda8806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda8806b810 0x7fda8815c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8815d640 0x7fda88159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda880cce70 con 0x7fda8806b430 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda880cccf0 con 0x7fda8815d640 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8fb62640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fda8815e200 con 0x7fda8806b810 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8d8fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda8806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8d8fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda8806ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55850/0 (socket says 172.21.15.6:55850) 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.071+0000 7fda8d8fe640 1 -- 172.21.15.6:0/248883487 learned_addr learned my addr 172.21.15.6:0/248883487 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8d8fe640 1 -- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda8806b810 msgr2=0x7fda8815c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8d8fe640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda8806b810 0x7fda8815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8d8fe640 1 -- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8815d640 msgr2=0x7fda88159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8d8fe640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8815d640 0x7fda88159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8d8fe640 1 -- 172.21.15.6:0/248883487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda880c5270 con 0x7fda8806b430 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8d8fe640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda8806ed10 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fda840098c0 tx=0x7fda840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda7e7fc640 1 -- 172.21.15.6:0/248883487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8403aa70 con 0x7fda8806b430 2024-03-18T14:43:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda7e7fc640 1 -- 172.21.15.6:0/248883487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8403ac10 con 0x7fda8806b430 2024-03-18T14:43:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda7e7fc640 1 -- 172.21.15.6:0/248883487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840443f0 con 0x7fda8806b430 2024-03-18T14:43:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda880c46f0 con 0x7fda8806b430 2024-03-18T14:43:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.072+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda880c4cb0 con 0x7fda8806b430 2024-03-18T14:43:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.073+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda880751b0 con 0x7fda8806b430 2024-03-18T14:43:57.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.073+0000 7fda7e7fc640 1 -- 172.21.15.6:0/248883487 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fda8404b060 con 0x7fda8806b430 2024-03-18T14:43:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.074+0000 7fda7e7fc640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fda6c06bbd0 0x7fda6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.074+0000 7fda7e7fc640 1 -- 172.21.15.6:0/248883487 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fda840ac980 con 0x7fda8806b430 2024-03-18T14:43:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.074+0000 7fda8d0fd640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fda6c06bbd0 0x7fda6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.075+0000 7fda8d0fd640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fda6c06bbd0 0x7fda6c06e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fda78007fd0 tx=0x7fda78007960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.078+0000 7fda7e7fc640 1 -- 172.21.15.6:0/248883487 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fda8403f070 con 0x7fda8806b430 2024-03-18T14:43:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.410+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fda880d1380 con 0x7fda8806b430 2024-03-18T14:43:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.410+0000 7fda7e7fc640 1 -- 172.21.15.6:0/248883487 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7fda8402f2e0 con 0x7fda8806b430 2024-03-18T14:43:57.412 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fda6c06bbd0 msgr2=0x7fda6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fda6c06bbd0 0x7fda6c06e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fda78007fd0 tx=0x7fda78007960 comp rx=0 tx=0).stop 2024-03-18T14:43:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 msgr2=0x7fda8806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda8806ed10 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fda840098c0 tx=0x7fda840046c0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 shutdown_connections 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fda6c06bbd0 0x7fda6c06e090 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8815d640 0x7fda88159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda8806b810 0x7fda8815c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 --2- 172.21.15.6:0/248883487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda8806b430 0x7fda8806ed10 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 >> 172.21.15.6:0/248883487 conn(0x7fda88099a50 msgr2=0x7fda880b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 shutdown_connections 2024-03-18T14:43:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.413+0000 7fda8fb62640 1 -- 172.21.15.6:0/248883487 wait complete. 2024-03-18T14:43:57.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:57.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.526+0000 7fb877880640 1 Processor -- start 2024-03-18T14:43:57.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.526+0000 7fb877880640 1 -- start start 2024-03-18T14:43:57.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb877880640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb8700cf080 0x7fb8700d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb877880640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb870161a10 0x7fb8700d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb877880640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8700d20e0 con 0x7fb87015dfb0 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb877880640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb87015e390 con 0x7fb870161a10 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb877880640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fb870161f40 con 0x7fb8700cf080 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb875e1d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb870161a10 0x7fb8700d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb875e1d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb870161a10 0x7fb8700d2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55852/0 (socket says 172.21.15.6:55852) 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb875e1d640 1 -- 172.21.15.6:0/4026776720 learned_addr learned my addr 172.21.15.6:0/4026776720 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb874e1b640 1 --2- 172.21.15.6:0/4026776720 >> v2:172.21.15.92:3300/0 conn(0x7fb8700cf080 0x7fb8700d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb875e1d640 1 -- 172.21.15.6:0/4026776720 >> v2:172.21.15.92:3300/0 conn(0x7fb8700cf080 msgr2=0x7fb8700d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb875e1d640 1 --2- 172.21.15.6:0/4026776720 >> v2:172.21.15.92:3300/0 conn(0x7fb8700cf080 0x7fb8700d25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb875e1d640 1 -- 172.21.15.6:0/4026776720 >> v1:172.21.15.92:6789/0 conn(0x7fb87015dfb0 legacy=0x7fb8700ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.527+0000 7fb875e1d640 1 -- 172.21.15.6:0/4026776720 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb870162220 con 0x7fb870161a10 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.528+0000 7fb875e1d640 1 --2- 172.21.15.6:0/4026776720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb870161a10 0x7fb8700d2b00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb8700d1660 tx=0x7fb86002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c0cf5791d6734e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb86ffff640 1 -- 172.21.15.6:0/4026776720 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860034070 con 0x7fb870161a10 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb86ffff640 1 -- 172.21.15.6:0/4026776720 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb860030860 con 0x7fb870161a10 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb86ffff640 1 -- 172.21.15.6:0/4026776720 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860030c30 con 0x7fb870161a10 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 -- 172.21.15.6:0/4026776720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb870161a10 msgr2=0x7fb8700d2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 --2- 172.21.15.6:0/4026776720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb870161a10 0x7fb8700d2b00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb8700d1660 tx=0x7fb86002b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 -- 172.21.15.6:0/4026776720 shutdown_connections 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 --2- 172.21.15.6:0/4026776720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb870161a10 0x7fb8700d2b00 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 --2- 172.21.15.6:0/4026776720 >> v2:172.21.15.92:3300/0 conn(0x7fb8700cf080 0x7fb8700d25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 -- 172.21.15.6:0/4026776720 >> 172.21.15.6:0/4026776720 conn(0x7fb87006c850 msgr2=0x7fb87006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 -- 172.21.15.6:0/4026776720 shutdown_connections 2024-03-18T14:43:57.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.529+0000 7fb877880640 1 -- 172.21.15.6:0/4026776720 wait complete. 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.530+0000 7fb877880640 1 Processor -- start 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb877880640 1 -- start start 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb877880640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8700cf080 0x7fb87014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb877880640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87015dfb0 0x7fb8700cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb877880640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb870161a10 0x7fb87015c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb877880640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb8700d1010 con 0x7fb87015dfb0 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb877880640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb8701627e0 con 0x7fb870161a10 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb877880640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fb8700d1190 con 0x7fb8700cf080 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb874e1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87015dfb0 0x7fb8700cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb874e1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87015dfb0 0x7fb8700cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55866/0 (socket says 172.21.15.6:55866) 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.531+0000 7fb874e1b640 1 -- 172.21.15.6:0/1885955627 learned_addr learned my addr 172.21.15.6:0/1885955627 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:57.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb875e1d640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb870161a10 0x7fb87015c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb87561c640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8700cf080 0x7fb87014e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb874e1b640 1 -- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8700cf080 msgr2=0x7fb87014e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb874e1b640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8700cf080 0x7fb87014e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb874e1b640 1 -- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb870161a10 msgr2=0x7fb87015c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb874e1b640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb870161a10 0x7fb87015c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb874e1b640 1 -- 172.21.15.6:0/1885955627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8700cc230 con 0x7fb87015dfb0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb87561c640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8700cf080 0x7fb87014e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb875e1d640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb870161a10 0x7fb87015c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb874e1b640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87015dfb0 0x7fb8700cbcf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb86400e9e0 tx=0x7fb86400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb86dffb640 1 -- 172.21.15.6:0/1885955627 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb864017070 con 0x7fb87015dfb0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb86dffb640 1 -- 172.21.15.6:0/1885955627 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86400c9d0 con 0x7fb87015dfb0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb86dffb640 1 -- 172.21.15.6:0/1885955627 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86400cdc0 con 0x7fb87015dfb0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8701f7a80 con 0x7fb87015dfb0 2024-03-18T14:43:57.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.532+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8701f8040 con 0x7fb87015dfb0 2024-03-18T14:43:57.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.533+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87004f5b0 con 0x7fb87015dfb0 2024-03-18T14:43:57.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.534+0000 7fb86dffb640 1 -- 172.21.15.6:0/1885955627 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb86401a070 con 0x7fb87015dfb0 2024-03-18T14:43:57.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.534+0000 7fb86dffb640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb85c06bbd0 0x7fb85c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.534+0000 7fb86dffb640 1 -- 172.21.15.6:0/1885955627 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fb86408ac90 con 0x7fb87015dfb0 2024-03-18T14:43:57.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.534+0000 7fb87561c640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb85c06bbd0 0x7fb85c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.535+0000 7fb87561c640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb85c06bbd0 0x7fb85c06e090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb87015d3b0 tx=0x7fb868009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.538+0000 7fb86dffb640 1 -- 172.21.15.6:0/1885955627 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb8640595f0 con 0x7fb87015dfb0 2024-03-18T14:43:57.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.780+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb8700688b0 con 0x7fb87015dfb0 2024-03-18T14:43:57.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.780+0000 7fb86dffb640 1 -- 172.21.15.6:0/1885955627 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7fb864058ef0 con 0x7fb87015dfb0 2024-03-18T14:43:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb85c06bbd0 msgr2=0x7fb85c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb85c06bbd0 0x7fb85c06e090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb87015d3b0 tx=0x7fb868009040 comp rx=0 tx=0).stop 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87015dfb0 msgr2=0x7fb8700cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87015dfb0 0x7fb8700cbcf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb86400e9e0 tx=0x7fb86400eea0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 shutdown_connections 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fb85c06bbd0 0x7fb85c06e090 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb870161a10 0x7fb87015c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87015dfb0 0x7fb8700cbcf0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 --2- 172.21.15.6:0/1885955627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8700cf080 0x7fb87014e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 >> 172.21.15.6:0/1885955627 conn(0x7fb87006c850 msgr2=0x7fb8700d2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 shutdown_connections 2024-03-18T14:43:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.782+0000 7fb877880640 1 -- 172.21.15.6:0/1885955627 wait complete. 2024-03-18T14:43:57.799 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:43:57.799 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:43:57.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:43:57.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.953+0000 7fd53709a640 1 Processor -- start 2024-03-18T14:43:57.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.953+0000 7fd53709a640 1 -- start start 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.953+0000 7fd53709a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd53015bdd0 0x7fd53006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.953+0000 7fd53709a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5300b3800 con 0x7fd5300cd7f0 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.953+0000 7fd53709a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5300cdc00 con 0x7fd53006b700 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.953+0000 7fd53709a640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd5300cded0 con 0x7fd53015bdd0 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.954+0000 7fd534e36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd53015bdd0 0x7fd53006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.954+0000 7fd535637640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd5300cd7f0 0x7fd5300b49d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35840/0 (socket says 172.21.15.6:35840) 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.954+0000 7fd535637640 1 -- 172.21.15.6:0/1583710527 learned_addr learned my addr 172.21.15.6:0/1583710527 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:57.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.954+0000 7fd534e36640 1 -- 172.21.15.6:0/1583710527 >> v1:172.21.15.92:6789/0 conn(0x7fd53006b700 legacy=0x7fd5300b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.954+0000 7fd534e36640 1 -- 172.21.15.6:0/1583710527 >> v1:172.21.15.6:6789/0 conn(0x7fd5300cd7f0 legacy=0x7fd5300b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.954+0000 7fd534e36640 1 -- 172.21.15.6:0/1583710527 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5300ce1b0 con 0x7fd53015bdd0 2024-03-18T14:43:57.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.954+0000 7fd52f7fe640 1 -- 172.21.15.6:0/1583710527 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260469374 0 0) 0x7fd5300b3800 con 0x7fd5300cd7f0 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd534e36640 1 --2- 172.21.15.6:0/1583710527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd53015bdd0 0x7fd53006b1c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd5240097a0 tx=0x7fd52402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c079f5480d4006 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd52f7fe640 1 -- 172.21.15.6:0/1583710527 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5240315a0 con 0x7fd53015bdd0 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd52f7fe640 1 -- 172.21.15.6:0/1583710527 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd524037070 con 0x7fd53015bdd0 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd52f7fe640 1 -- 172.21.15.6:0/1583710527 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524031960 con 0x7fd53015bdd0 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd53709a640 1 -- 172.21.15.6:0/1583710527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd53015bdd0 msgr2=0x7fd53006b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd53709a640 1 --2- 172.21.15.6:0/1583710527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd53015bdd0 0x7fd53006b1c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd5240097a0 tx=0x7fd52402b840 comp rx=0 tx=0).stop 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd53709a640 1 -- 172.21.15.6:0/1583710527 shutdown_connections 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd53709a640 1 --2- 172.21.15.6:0/1583710527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd53015bdd0 0x7fd53006b1c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd53709a640 1 -- 172.21.15.6:0/1583710527 >> 172.21.15.6:0/1583710527 conn(0x7fd5300d09b0 msgr2=0x7fd5300d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd53709a640 1 -- 172.21.15.6:0/1583710527 shutdown_connections 2024-03-18T14:43:57.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.955+0000 7fd53709a640 1 -- 172.21.15.6:0/1583710527 wait complete. 2024-03-18T14:43:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.957+0000 7fd53709a640 1 Processor -- start 2024-03-18T14:43:57.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.957+0000 7fd53709a640 1 -- start start 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd53709a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd53006b700 0x7fd530152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd53709a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5300cd7f0 0x7fd5300af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd53709a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd53015bdd0 0x7fd5300afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd53709a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd53015c1b0 con 0x7fd5300cd7f0 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd53709a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd5300b3da0 con 0x7fd53015bdd0 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd53709a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd5300cb190 con 0x7fd53006b700 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5300cd7f0 0x7fd5300af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5300cd7f0 0x7fd5300af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55884/0 (socket says 172.21.15.6:55884) 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 -- 172.21.15.6:0/990951286 learned_addr learned my addr 172.21.15.6:0/990951286 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd535637640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd53015bdd0 0x7fd5300afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd534e36640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd53006b700 0x7fd530152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 -- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd53006b700 msgr2=0x7fd530152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd53006b700 0x7fd530152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 -- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd53015bdd0 msgr2=0x7fd5300afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd53015bdd0 0x7fd5300afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 -- 172.21.15.6:0/990951286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5300ae3e0 con 0x7fd5300cd7f0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52ffff640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5300cd7f0 0x7fd5300af910 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c430 tx=0x7fd51c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52d7fa640 1 -- 172.21.15.6:0/990951286 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c011790 con 0x7fd5300cd7f0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.959+0000 7fd52d7fa640 1 -- 172.21.15.6:0/990951286 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd51c017070 con 0x7fd5300cd7f0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.960+0000 7fd52d7fa640 1 -- 172.21.15.6:0/990951286 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c011b70 con 0x7fd5300cd7f0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.960+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5300ae6c0 con 0x7fd5300cd7f0 2024-03-18T14:43:57.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.960+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5300aec80 con 0x7fd5300cd7f0 2024-03-18T14:43:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.961+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd53009e9a0 con 0x7fd5300cd7f0 2024-03-18T14:43:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.961+0000 7fd52d7fa640 1 -- 172.21.15.6:0/990951286 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd51c011d10 con 0x7fd5300cd7f0 2024-03-18T14:43:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.962+0000 7fd52d7fa640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd51406bbd0 0x7fd51406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.962+0000 7fd52d7fa640 1 -- 172.21.15.6:0/990951286 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fd51c08a910 con 0x7fd5300cd7f0 2024-03-18T14:43:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.962+0000 7fd534e36640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd51406bbd0 0x7fd51406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.962+0000 7fd534e36640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd51406bbd0 0x7fd51406e090 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd524036000 tx=0x7fd52403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:57.966+0000 7fd52d7fa640 1 -- 172.21.15.6:0/990951286 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd51c058c20 con 0x7fd5300cd7f0 2024-03-18T14:43:58.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.155+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd53004f620 con 0x7fd5300cd7f0 2024-03-18T14:43:58.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.156+0000 7fd52d7fa640 1 -- 172.21.15.6:0/990951286 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7fd51c05ccc0 con 0x7fd5300cd7f0 2024-03-18T14:43:58.159 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:43:58.159 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:43:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd51406bbd0 msgr2=0x7fd51406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd51406bbd0 0x7fd51406e090 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd524036000 tx=0x7fd52403d040 comp rx=0 tx=0).stop 2024-03-18T14:43:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5300cd7f0 msgr2=0x7fd5300af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5300cd7f0 0x7fd5300af910 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c430 tx=0x7fd51c00c8f0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 shutdown_connections 2024-03-18T14:43:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd51406bbd0 0x7fd51406e090 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd53015bdd0 0x7fd5300afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5300cd7f0 0x7fd5300af910 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 --2- 172.21.15.6:0/990951286 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd53006b700 0x7fd530152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 >> 172.21.15.6:0/990951286 conn(0x7fd5300d09b0 msgr2=0x7fd530156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 shutdown_connections 2024-03-18T14:43:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.158+0000 7fd53709a640 1 -- 172.21.15.6:0/990951286 wait complete. 2024-03-18T14:43:58.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:43:39.597879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:43:40.299491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:43:58.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:43:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.274+0000 7f0c81636640 1 Processor -- start 2024-03-18T14:43:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.275+0000 7f0c81636640 1 -- start start 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.275+0000 7f0c81636640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c15e030 0x7f0c7c0d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.275+0000 7f0c81636640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0c7c0d0450 0x7f0c7c0d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.275+0000 7f0c81636640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c7c0d0e20 con 0x7f0c7c0b6140 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.275+0000 7f0c81636640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0c7c15e410 con 0x7f0c7c0d0450 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.275+0000 7f0c81636640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7c0ce300 con 0x7f0c7c15e030 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7affd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c15e030 0x7f0c7c0d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7b7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0c7c0d0450 0x7f0c7c0d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7affd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c15e030 0x7f0c7c0d3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36758/0 (socket says 172.21.15.6:36758) 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7b7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0c7c0d0450 0x7f0c7c0d08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:45994/0 (socket says 172.21.15.6:45994) 2024-03-18T14:43:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7affd640 1 -- 172.21.15.6:0/1246542074 learned_addr learned my addr 172.21.15.6:0/1246542074 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7affd640 1 -- 172.21.15.6:0/1246542074 >> v2:172.21.15.92:3300/0 conn(0x7f0c7c0d0450 msgr2=0x7f0c7c0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7affd640 1 --2- 172.21.15.6:0/1246542074 >> v2:172.21.15.92:3300/0 conn(0x7f0c7c0d0450 0x7f0c7c0d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7affd640 1 -- 172.21.15.6:0/1246542074 >> v1:172.21.15.92:6789/0 conn(0x7f0c7c0b6140 legacy=0x7f0c7c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7affd640 1 -- 172.21.15.6:0/1246542074 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c0b65a0 con 0x7f0c7c15e030 2024-03-18T14:43:58.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.276+0000 7f0c7b7fe640 1 --2- 172.21.15.6:0/1246542074 >> v2:172.21.15.92:3300/0 conn(0x7f0c7c0d0450 0x7f0c7c0d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.277+0000 7f0c7affd640 1 --2- 172.21.15.6:0/1246542074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c15e030 0x7f0c7c0d3570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009870 tx=0x7f0c6c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4837ab86dcdeb90d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.277+0000 7f0c7a7fc640 1 -- 172.21.15.6:0/1246542074 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c034070 con 0x7f0c7c15e030 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.277+0000 7f0c7a7fc640 1 -- 172.21.15.6:0/1246542074 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c6c030840 con 0x7f0c7c15e030 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.277+0000 7f0c7a7fc640 1 -- 172.21.15.6:0/1246542074 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c030c10 con 0x7f0c7c15e030 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 -- 172.21.15.6:0/1246542074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c15e030 msgr2=0x7f0c7c0d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 --2- 172.21.15.6:0/1246542074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c15e030 0x7f0c7c0d3570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009870 tx=0x7f0c6c02b980 comp rx=0 tx=0).stop 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 -- 172.21.15.6:0/1246542074 shutdown_connections 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 --2- 172.21.15.6:0/1246542074 >> v2:172.21.15.92:3300/0 conn(0x7f0c7c0d0450 0x7f0c7c0d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 --2- 172.21.15.6:0/1246542074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c15e030 0x7f0c7c0d3570 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 -- 172.21.15.6:0/1246542074 >> 172.21.15.6:0/1246542074 conn(0x7f0c7c06c8c0 msgr2=0x7f0c7c06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 -- 172.21.15.6:0/1246542074 shutdown_connections 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.278+0000 7f0c81636640 1 -- 172.21.15.6:0/1246542074 wait complete. 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.279+0000 7f0c81636640 1 Processor -- start 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.279+0000 7f0c81636640 1 -- start start 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c81636640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c7c0b6140 0x7f0c7c161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c81636640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c0d0450 0x7f0c7c15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c81636640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c7c15e030 0x7f0c7c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c81636640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c0ce550 con 0x7f0c7c15e030 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c81636640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c0d3290 con 0x7f0c7c0b6140 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c81636640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7c0ce830 con 0x7f0c7c0d0450 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c7c15e030 0x7f0c7c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c7c15e030 0x7f0c7c158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55896/0 (socket says 172.21.15.6:55896) 2024-03-18T14:43:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 -- 172.21.15.6:0/4028594385 learned_addr learned my addr 172.21.15.6:0/4028594385 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7affd640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c7c0b6140 0x7f0c7c161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c73fff640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c0d0450 0x7f0c7c15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 -- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c0d0450 msgr2=0x7f0c7c15efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c0d0450 0x7f0c7c15efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 -- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c7c0b6140 msgr2=0x7f0c7c161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c7c0b6140 0x7f0c7c161110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 -- 172.21.15.6:0/4028594385 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c15f7b0 con 0x7f0c7c15e030 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c7b7fe640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c7c15e030 0x7f0c7c158d80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400efc0 tx=0x7f0c74009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c737fe640 1 -- 172.21.15.6:0/4028594385 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74018030 con 0x7f0c7c15e030 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c737fe640 1 -- 172.21.15.6:0/4028594385 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c7400cde0 con 0x7f0c7c15e030 2024-03-18T14:43:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7c15f4e0 con 0x7f0c7c15e030 2024-03-18T14:43:58.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.280+0000 7f0c737fe640 1 -- 172.21.15.6:0/4028594385 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74003bb0 con 0x7f0c7c15e030 2024-03-18T14:43:58.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.281+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7c15ea10 con 0x7f0c7c15e030 2024-03-18T14:43:58.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.281+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c7c0751b0 con 0x7f0c7c15e030 2024-03-18T14:43:58.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.282+0000 7f0c737fe640 1 -- 172.21.15.6:0/4028594385 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0c74005d10 con 0x7f0c7c15e030 2024-03-18T14:43:58.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.282+0000 7f0c737fe640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0c5c06bbd0 0x7f0c5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.282+0000 7f0c737fe640 1 -- 172.21.15.6:0/4028594385 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f0c7408b4a0 con 0x7f0c7c15e030 2024-03-18T14:43:58.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.283+0000 7f0c7affd640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0c5c06bbd0 0x7f0c5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.283+0000 7f0c7affd640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0c5c06bbd0 0x7f0c5c06e090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c036000 tx=0x7f0c6c03b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.291+0000 7f0c737fe640 1 -- 172.21.15.6:0/4028594385 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0c74022070 con 0x7f0c7c15e030 2024-03-18T14:43:58.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.473+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0c7c0d19f0 con 0x7f0c7c15e030 2024-03-18T14:43:58.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.473+0000 7f0c737fe640 1 -- 172.21.15.6:0/4028594385 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f0c7c15f7b0 con 0x7f0c7c15e030 2024-03-18T14:43:58.475 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0c5c06bbd0 msgr2=0x7f0c5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0c5c06bbd0 0x7f0c5c06e090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c036000 tx=0x7f0c6c03b040 comp rx=0 tx=0).stop 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c7c15e030 msgr2=0x7f0c7c158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c7c15e030 0x7f0c7c158d80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400efc0 tx=0x7f0c74009f90 comp rx=0 tx=0).stop 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 shutdown_connections 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0c5c06bbd0 0x7f0c5c06e090 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c7c15e030 0x7f0c7c158d80 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c7c0d0450 0x7f0c7c15efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 --2- 172.21.15.6:0/4028594385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c7c0b6140 0x7f0c7c161110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 >> 172.21.15.6:0/4028594385 conn(0x7f0c7c06c8c0 msgr2=0x7f0c7c0b8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 shutdown_connections 2024-03-18T14:43:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.475+0000 7f0c81636640 1 -- 172.21.15.6:0/4028594385 wait complete. 2024-03-18T14:43:58.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:43:58.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.591+0000 7f85bed0c640 1 Processor -- start 2024-03-18T14:43:58.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.591+0000 7f85bed0c640 1 -- start start 2024-03-18T14:43:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bed0c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f85b815e030 0x7f85b80d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bed0c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80b6140 0x7f85b80ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bed0c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f85b80d0450 0x7f85b80d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bed0c640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f85b815e410 con 0x7f85b80d0450 2024-03-18T14:43:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bed0c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f85b80ce300 con 0x7f85b815e030 2024-03-18T14:43:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bed0c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f85b80d0e20 con 0x7f85b80b6140 2024-03-18T14:43:58.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bd2a9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f85b80d0450 0x7f85b80d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bd2a9640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f85b80d0450 0x7f85b80d08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55908/0 (socket says 172.21.15.6:55908) 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.592+0000 7f85bd2a9640 1 -- 172.21.15.6:0/111195796 learned_addr learned my addr 172.21.15.6:0/111195796 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85b7fff640 1 --2- 172.21.15.6:0/111195796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80b6140 0x7f85b80ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85bcaa8640 1 --2- 172.21.15.6:0/111195796 >> v2:172.21.15.92:3300/0 conn(0x7f85b815e030 0x7f85b80d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85bd2a9640 1 -- 172.21.15.6:0/111195796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80b6140 msgr2=0x7f85b80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85bd2a9640 1 --2- 172.21.15.6:0/111195796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80b6140 0x7f85b80ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85bd2a9640 1 -- 172.21.15.6:0/111195796 >> v2:172.21.15.92:3300/0 conn(0x7f85b815e030 msgr2=0x7f85b80d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85bd2a9640 1 --2- 172.21.15.6:0/111195796 >> v2:172.21.15.92:3300/0 conn(0x7f85b815e030 0x7f85b80d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85bd2a9640 1 -- 172.21.15.6:0/111195796 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b80b65a0 con 0x7f85b80d0450 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.593+0000 7f85bd2a9640 1 --2- 172.21.15.6:0/111195796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80d0450 0x7f85b80d08e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f85b8068ad0 tx=0x7f85b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff856a43a1f55be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85b77fe640 1 -- 172.21.15.6:0/111195796 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b002ca70 con 0x7f85b80d0450 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85b77fe640 1 -- 172.21.15.6:0/111195796 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b002cc10 con 0x7f85b80d0450 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85b77fe640 1 -- 172.21.15.6:0/111195796 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b00343f0 con 0x7f85b80d0450 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 -- 172.21.15.6:0/111195796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80d0450 msgr2=0x7f85b80d08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 --2- 172.21.15.6:0/111195796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80d0450 0x7f85b80d08e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f85b8068ad0 tx=0x7f85b002bed0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 -- 172.21.15.6:0/111195796 shutdown_connections 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 --2- 172.21.15.6:0/111195796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80d0450 0x7f85b80d08e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 --2- 172.21.15.6:0/111195796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80b6140 0x7f85b80ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 --2- 172.21.15.6:0/111195796 >> v2:172.21.15.92:3300/0 conn(0x7f85b815e030 0x7f85b80d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 -- 172.21.15.6:0/111195796 >> 172.21.15.6:0/111195796 conn(0x7f85b806c8c0 msgr2=0x7f85b806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 -- 172.21.15.6:0/111195796 shutdown_connections 2024-03-18T14:43:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.594+0000 7f85bed0c640 1 -- 172.21.15.6:0/111195796 wait complete. 2024-03-18T14:43:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.595+0000 7f85bed0c640 1 Processor -- start 2024-03-18T14:43:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.595+0000 7f85bed0c640 1 -- start start 2024-03-18T14:43:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bed0c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80b6140 0x7f85b814cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bed0c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80d0450 0x7f85b815fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bed0c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85b815e030 0x7f85b81602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bed0c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f85b80ce550 con 0x7f85b80b6140 2024-03-18T14:43:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bed0c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85b80d3290 con 0x7f85b815e030 2024-03-18T14:43:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bed0c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f85b80ce830 con 0x7f85b80d0450 2024-03-18T14:43:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bcaa8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80b6140 0x7f85b814cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bcaa8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80b6140 0x7f85b814cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55920/0 (socket says 172.21.15.6:55920) 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.596+0000 7f85bcaa8640 1 -- 172.21.15.6:0/3093329946 learned_addr learned my addr 172.21.15.6:0/3093329946 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bd2a9640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85b815e030 0x7f85b81602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85b7fff640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80d0450 0x7f85b815fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bcaa8640 1 -- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80d0450 msgr2=0x7f85b815fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bcaa8640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80d0450 0x7f85b815fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bcaa8640 1 -- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85b815e030 msgr2=0x7f85b81602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bcaa8640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85b815e030 0x7f85b81602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bcaa8640 1 -- 172.21.15.6:0/3093329946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b80cdb90 con 0x7f85b80b6140 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85b7fff640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80d0450 0x7f85b815fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bcaa8640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80b6140 0x7f85b814cd90 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00d5f0 tx=0x7f85ac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85b57fa640 1 -- 172.21.15.6:0/3093329946 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0125f0 con 0x7f85b80b6140 2024-03-18T14:43:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85b57fa640 1 -- 172.21.15.6:0/3093329946 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85ac019070 con 0x7f85b80b6140 2024-03-18T14:43:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b80cb140 con 0x7f85b80b6140 2024-03-18T14:43:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85b57fa640 1 -- 172.21.15.6:0/3093329946 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0129d0 con 0x7f85b80b6140 2024-03-18T14:43:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.597+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b8160f50 con 0x7f85b80b6140 2024-03-18T14:43:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.598+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b80751b0 con 0x7f85b80b6140 2024-03-18T14:43:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.599+0000 7f85b57fa640 1 -- 172.21.15.6:0/3093329946 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f85ac012b70 con 0x7f85b80b6140 2024-03-18T14:43:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.599+0000 7f85b57fa640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f859c06bbd0 0x7f859c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.599+0000 7f85b57fa640 1 -- 172.21.15.6:0/3093329946 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f85ac08af20 con 0x7f85b80b6140 2024-03-18T14:43:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.600+0000 7f85b7fff640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f859c06bbd0 0x7f859c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.600+0000 7f85b7fff640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f859c06bbd0 0x7f859c06e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f85b814d9f0 tx=0x7f85a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.604+0000 7f85b57fa640 1 -- 172.21.15.6:0/3093329946 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f85ac059910 con 0x7f85b80b6140 2024-03-18T14:43:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.788+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f85b8161800 con 0x7f85b80b6140 2024-03-18T14:43:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.789+0000 7f85b57fa640 1 -- 172.21.15.6:0/3093329946 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f85ac059210 con 0x7f85b80b6140 2024-03-18T14:43:58.791 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f859c06bbd0 msgr2=0x7f859c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f859c06bbd0 0x7f859c06e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f85b814d9f0 tx=0x7f85a4008040 comp rx=0 tx=0).stop 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80b6140 msgr2=0x7f85b814cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80b6140 0x7f85b814cd90 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00d5f0 tx=0x7f85ac00dab0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 shutdown_connections 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f859c06bbd0 0x7f859c06e090 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85b815e030 0x7f85b81602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f85b80d0450 0x7f85b815fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 --2- 172.21.15.6:0/3093329946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f85b80b6140 0x7f85b814cd90 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 >> 172.21.15.6:0/3093329946 conn(0x7f85b806c8c0 msgr2=0x7f85b80b6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:58.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 shutdown_connections 2024-03-18T14:43:58.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.791+0000 7f85bed0c640 1 -- 172.21.15.6:0/3093329946 wait complete. 2024-03-18T14:43:58.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:43:58.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.907+0000 7f7db2d0c640 1 Processor -- start 2024-03-18T14:43:58.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.907+0000 7f7db2d0c640 1 -- start start 2024-03-18T14:43:58.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.907+0000 7f7db2d0c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7dac15e030 0x7f7dac0d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.907+0000 7f7db2d0c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac0b6140 0x7f7dac0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.907+0000 7f7db2d0c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dac0b65a0 con 0x7f7dac0d0450 2024-03-18T14:43:58.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.907+0000 7f7db2d0c640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7dac15e410 con 0x7f7dac15e030 2024-03-18T14:43:58.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.907+0000 7f7db2d0c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7dac0ce300 con 0x7f7dac0b6140 2024-03-18T14:43:58.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7db0aa8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7dac15e030 0x7f7dac0d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7db12a9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7dac0d0450 0x7f7dac0d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35856/0 (socket says 172.21.15.6:35856) 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7db12a9640 1 -- 172.21.15.6:0/20072651 learned_addr learned my addr 172.21.15.6:0/20072651 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7dabfff640 1 --2- 172.21.15.6:0/20072651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac0b6140 0x7f7dac0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7db0aa8640 1 -- 172.21.15.6:0/20072651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac0b6140 msgr2=0x7f7dac0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7db0aa8640 1 --2- 172.21.15.6:0/20072651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac0b6140 0x7f7dac0ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7db0aa8640 1 -- 172.21.15.6:0/20072651 >> v1:172.21.15.6:6789/0 conn(0x7f7dac0d0450 legacy=0x7f7dac0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7db0aa8640 1 -- 172.21.15.6:0/20072651 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac0d2fd0 con 0x7f7dac15e030 2024-03-18T14:43:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.908+0000 7f7dab7fe640 1 -- 172.21.15.6:0/20072651 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2579259424 0 0) 0x7f7dac0b65a0 con 0x7f7dac0d0450 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.909+0000 7f7db0aa8640 1 --2- 172.21.15.6:0/20072651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15e030 0x7f7dac0d3570 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009d90 tx=0x7f7da402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4792c34ace3e97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.909+0000 7f7dab7fe640 1 -- 172.21.15.6:0/20072651 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da4039070 con 0x7f7dac15e030 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.909+0000 7f7dab7fe640 1 -- 172.21.15.6:0/20072651 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da402cc50 con 0x7f7dac15e030 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.909+0000 7f7dab7fe640 1 -- 172.21.15.6:0/20072651 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da4034440 con 0x7f7dac15e030 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.909+0000 7f7db2d0c640 1 -- 172.21.15.6:0/20072651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15e030 msgr2=0x7f7dac0d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.909+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/20072651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15e030 0x7f7dac0d3570 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009d90 tx=0x7f7da402bea0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.910+0000 7f7db2d0c640 1 -- 172.21.15.6:0/20072651 shutdown_connections 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.910+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/20072651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac0b6140 0x7f7dac0ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.910+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/20072651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15e030 0x7f7dac0d3570 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.910+0000 7f7db2d0c640 1 -- 172.21.15.6:0/20072651 >> 172.21.15.6:0/20072651 conn(0x7f7dac06c8c0 msgr2=0x7f7dac06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.910+0000 7f7db2d0c640 1 -- 172.21.15.6:0/20072651 shutdown_connections 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.910+0000 7f7db2d0c640 1 -- 172.21.15.6:0/20072651 wait complete. 2024-03-18T14:43:58.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.911+0000 7f7db2d0c640 1 Processor -- start 2024-03-18T14:43:58.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.911+0000 7f7db2d0c640 1 -- start start 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.911+0000 7f7db2d0c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac0b6140 0x7f7dac15f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db2d0c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7dac0d0450 0x7f7dac1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db2d0c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac15e030 0x7f7dac15a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db2d0c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac0ce830 con 0x7f7dac0b6140 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db2d0c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac0ce550 con 0x7f7dac0d0450 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db2d0c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7dac0cf400 con 0x7f7dac15e030 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac0b6140 0x7f7dac15f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac0b6140 0x7f7dac15f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55936/0 (socket says 172.21.15.6:55936) 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 -- 172.21.15.6:0/221822756 learned_addr learned my addr 172.21.15.6:0/221822756 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7dabfff640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7dac0d0450 0x7f7dac1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 -- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac15e030 msgr2=0x7f7dac15a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db12a9640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac15e030 0x7f7dac15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac15e030 0x7f7dac15a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 -- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7dac0d0450 msgr2=0x7f7dac1f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7dac0d0450 0x7f7dac1f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 -- 172.21.15.6:0/221822756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac0cc470 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db0aa8640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac0b6140 0x7f7dac15f590 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009d90 tx=0x7f7da4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7da97fa640 1 -- 172.21.15.6:0/221822756 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da4039050 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7da97fa640 1 -- 172.21.15.6:0/221822756 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da403a040 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dac0cc710 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7da97fa640 1 -- 172.21.15.6:0/221822756 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da402ca50 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.912+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dac0cccd0 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.913+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dac0751b0 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.914+0000 7f7da97fa640 1 -- 172.21.15.6:0/221822756 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7da4047020 con 0x7f7dac0b6140 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.914+0000 7f7da97fa640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d9806bbd0 0x7f7d9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:58.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.914+0000 7f7da97fa640 1 -- 172.21.15.6:0/221822756 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f7da40ace80 con 0x7f7dac0b6140 2024-03-18T14:43:58.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.915+0000 7f7dabfff640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d9806bbd0 0x7f7d9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:58.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.915+0000 7f7dabfff640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d9806bbd0 0x7f7d9806e090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7da0006fd0 tx=0x7f7da0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:58.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:58.923+0000 7f7da97fa640 1 -- 172.21.15.6:0/221822756 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7da404e020 con 0x7f7dac0b6140 2024-03-18T14:43:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.104+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7dac0d19f0 con 0x7f7dac0b6140 2024-03-18T14:43:59.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.105+0000 7f7da97fa640 1 -- 172.21.15.6:0/221822756 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f7da407b740 con 0x7f7dac0b6140 2024-03-18T14:43:59.107 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:43:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d9806bbd0 msgr2=0x7f7d9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d9806bbd0 0x7f7d9806e090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7da0006fd0 tx=0x7f7da0008040 comp rx=0 tx=0).stop 2024-03-18T14:43:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac0b6140 msgr2=0x7f7dac15f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac0b6140 0x7f7dac15f590 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009d90 tx=0x7f7da4004a40 comp rx=0 tx=0).stop 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 shutdown_connections 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f7d9806bbd0 0x7f7d9806e090 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7dac15e030 0x7f7dac15a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7dac0d0450 0x7f7dac1f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 --2- 172.21.15.6:0/221822756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac0b6140 0x7f7dac15f590 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 >> 172.21.15.6:0/221822756 conn(0x7f7dac06c8c0 msgr2=0x7f7dac0d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 shutdown_connections 2024-03-18T14:43:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.107+0000 7f7db2d0c640 1 -- 172.21.15.6:0/221822756 wait complete. 2024-03-18T14:43:59.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:43:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.223+0000 7f22f747e640 1 Processor -- start 2024-03-18T14:43:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.224+0000 7f22f747e640 1 -- start start 2024-03-18T14:43:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.224+0000 7f22f747e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22f00d2fd0 0x7f22f00b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.224+0000 7f22f747e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f22f00b6df0 0x7f22f00b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.224+0000 7f22f747e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22f00d3430 con 0x7f22f006c7a0 2024-03-18T14:43:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.224+0000 7f22f747e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f22f00d1f30 con 0x7f22f00d2fd0 2024-03-18T14:43:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.224+0000 7f22f747e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f22f00d2210 con 0x7f22f00b6df0 2024-03-18T14:43:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.224+0000 7f22f4a19640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22f00d2fd0 0x7f22f00b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f4a19640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22f00d2fd0 0x7f22f00b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55948/0 (socket says 172.21.15.6:55948) 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f4a19640 1 -- 172.21.15.6:0/856690032 learned_addr learned my addr 172.21.15.6:0/856690032 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f5a1b640 1 --2- 172.21.15.6:0/856690032 >> v2:172.21.15.92:3300/0 conn(0x7f22f00b6df0 0x7f22f00b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f4a19640 1 -- 172.21.15.6:0/856690032 >> v2:172.21.15.92:3300/0 conn(0x7f22f00b6df0 msgr2=0x7f22f00b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f4a19640 1 --2- 172.21.15.6:0/856690032 >> v2:172.21.15.92:3300/0 conn(0x7f22f00b6df0 0x7f22f00b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f4a19640 1 -- 172.21.15.6:0/856690032 >> v1:172.21.15.92:6789/0 conn(0x7f22f006c7a0 legacy=0x7f22f006cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f4a19640 1 -- 172.21.15.6:0/856690032 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f00d24f0 con 0x7f22f00d2fd0 2024-03-18T14:43:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.225+0000 7f22f5a1b640 1 --2- 172.21.15.6:0/856690032 >> v2:172.21.15.92:3300/0 conn(0x7f22f00b6df0 0x7f22f00b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:43:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f4a19640 1 --2- 172.21.15.6:0/856690032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f00b95e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f22f00b6ba0 tx=0x7f22e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbc354693ae8fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22e7fff640 1 -- 172.21.15.6:0/856690032 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0034070 con 0x7f22f00d2fd0 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22e7fff640 1 -- 172.21.15.6:0/856690032 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e0030860 con 0x7f22f00d2fd0 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22e7fff640 1 -- 172.21.15.6:0/856690032 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0030c30 con 0x7f22f00d2fd0 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f747e640 1 -- 172.21.15.6:0/856690032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 msgr2=0x7f22f00b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f747e640 1 --2- 172.21.15.6:0/856690032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f00b95e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f22f00b6ba0 tx=0x7f22e002b9a0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f747e640 1 -- 172.21.15.6:0/856690032 shutdown_connections 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f747e640 1 --2- 172.21.15.6:0/856690032 >> v2:172.21.15.92:3300/0 conn(0x7f22f00b6df0 0x7f22f00b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f747e640 1 --2- 172.21.15.6:0/856690032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f00b95e0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f747e640 1 -- 172.21.15.6:0/856690032 >> 172.21.15.6:0/856690032 conn(0x7f22f0072ce0 msgr2=0x7f22f00730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.226+0000 7f22f747e640 1 -- 172.21.15.6:0/856690032 shutdown_connections 2024-03-18T14:43:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.227+0000 7f22f747e640 1 -- 172.21.15.6:0/856690032 wait complete. 2024-03-18T14:43:59.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 Processor -- start 2024-03-18T14:43:59.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 -- start start 2024-03-18T14:43:59.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22f006c7a0 0x7f22f015f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22f00b6df0 0x7f22f0158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f0160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f22f00b9b20 con 0x7f22f00d2fd0 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22f00d2ab0 con 0x7f22f00b6df0 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f747e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f22f00b9ca0 con 0x7f22f006c7a0 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f5a1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f0160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f5a1b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f0160800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55954/0 (socket says 172.21.15.6:55954) 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.228+0000 7f22f5a1b640 1 -- 172.21.15.6:0/2350592884 learned_addr learned my addr 172.21.15.6:0/2350592884 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f4a19640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22f00b6df0 0x7f22f0158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f521a640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22f006c7a0 0x7f22f015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f5a1b640 1 -- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22f006c7a0 msgr2=0x7f22f015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f5a1b640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22f006c7a0 0x7f22f015f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f5a1b640 1 -- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22f00b6df0 msgr2=0x7f22f0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f5a1b640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22f00b6df0 0x7f22f0158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f5a1b640 1 -- 172.21.15.6:0/2350592884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f015e880 con 0x7f22f00d2fd0 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f5a1b640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f0160800 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f22e8007ef0 tx=0x7f22e800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22e5ffb640 1 -- 172.21.15.6:0/2350592884 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e801a070 con 0x7f22f00d2fd0 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22e5ffb640 1 -- 172.21.15.6:0/2350592884 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e8003b90 con 0x7f22f00d2fd0 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f015eb60 con 0x7f22f00d2fd0 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22e5ffb640 1 -- 172.21.15.6:0/2350592884 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8004030 con 0x7f22f00d2fd0 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.229+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f015f120 con 0x7f22f00d2fd0 2024-03-18T14:43:59.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.230+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f00730c0 con 0x7f22f00d2fd0 2024-03-18T14:43:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.231+0000 7f22e5ffb640 1 -- 172.21.15.6:0/2350592884 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f22e801d070 con 0x7f22f00d2fd0 2024-03-18T14:43:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.231+0000 7f22e5ffb640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f22d406bbd0 0x7f22d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:43:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.231+0000 7f22e5ffb640 1 -- 172.21.15.6:0/2350592884 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f22e808ae40 con 0x7f22f00d2fd0 2024-03-18T14:43:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.232+0000 7f22f521a640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f22d406bbd0 0x7f22d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:43:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.232+0000 7f22f521a640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f22d406bbd0 0x7f22d406e090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f22ec004f60 tx=0x7f22ec007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:43:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.240+0000 7f22e5ffb640 1 -- 172.21.15.6:0/2350592884 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f22f00730c0 con 0x7f22f00d2fd0 2024-03-18T14:43:59.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.488+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f22f00b2d40 con 0x7f22f00d2fd0 2024-03-18T14:43:59.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.488+0000 7f22e5ffb640 1 -- 172.21.15.6:0/2350592884 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f22e8010070 con 0x7f22f00d2fd0 2024-03-18T14:43:59.490 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:43:59.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f22d406bbd0 msgr2=0x7f22d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f22d406bbd0 0x7f22d406e090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f22ec004f60 tx=0x7f22ec007e30 comp rx=0 tx=0).stop 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 msgr2=0x7f22f0160800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f0160800 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f22e8007ef0 tx=0x7f22e800d4a0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 shutdown_connections 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f22d406bbd0 0x7f22d406e090 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22f00d2fd0 0x7f22f0160800 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22f00b6df0 0x7f22f0158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 --2- 172.21.15.6:0/2350592884 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22f006c7a0 0x7f22f015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 >> 172.21.15.6:0/2350592884 conn(0x7f22f0072ce0 msgr2=0x7f22f00cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 shutdown_connections 2024-03-18T14:43:59.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:43:59.490+0000 7f22f747e640 1 -- 172.21.15.6:0/2350592884 wait complete. 2024-03-18T14:43:59.503 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:43:59.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:43:59.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T14:43:59.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[60883]: fuse finished with error 0 and tester_r 0 2024-03-18T14:43:59.531 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:05.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:05.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:05.632 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:05.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:05.633 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:44:05.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T14:44:05.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.792+0000 7ff3b22dc640 1 Processor -- start 2024-03-18T14:44:05.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.792+0000 7ff3b22dc640 1 -- start start 2024-03-18T14:44:05.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b22dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac15e020 0x7ff3ac162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b22dc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff3ac0ced00 0x7ff3ac0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b22dc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3ac0d3210 0x7ff3ac0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b22dc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff3ac0b9660 con 0x7ff3ac0d3210 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b22dc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7ff3ac0ce390 con 0x7ff3ac0ced00 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b22dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff3ac0b9900 con 0x7ff3ac15e020 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3ac0d3210 0x7ff3ac0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3ac0d3210 0x7ff3ac0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37764/0 (socket says 172.21.15.6:37764) 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 -- 172.21.15.6:0/1970496354 learned_addr learned my addr 172.21.15.6:0/1970496354 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3abfff640 1 --2- 172.21.15.6:0/1970496354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac15e020 0x7ff3ac162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3ab7fe640 1 --2- 172.21.15.6:0/1970496354 >> v2:172.21.15.92:3300/0 conn(0x7ff3ac0ced00 0x7ff3ac0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 -- 172.21.15.6:0/1970496354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac15e020 msgr2=0x7ff3ac162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 --2- 172.21.15.6:0/1970496354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac15e020 0x7ff3ac162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 -- 172.21.15.6:0/1970496354 >> v2:172.21.15.92:3300/0 conn(0x7ff3ac0ced00 msgr2=0x7ff3ac0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 --2- 172.21.15.6:0/1970496354 >> v2:172.21.15.92:3300/0 conn(0x7ff3ac0ced00 0x7ff3ac0d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.793+0000 7ff3b0879640 1 -- 172.21.15.6:0/1970496354 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ac0b9be0 con 0x7ff3ac0d3210 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.794+0000 7ff3b0879640 1 --2- 172.21.15.6:0/1970496354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac0d3210 0x7ff3ac0d3c30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0b6820 tx=0x7ff3a402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2bdcd90cb1a1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.794+0000 7ff3aaffd640 1 -- 172.21.15.6:0/1970496354 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a402c9e0 con 0x7ff3ac0d3210 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.794+0000 7ff3aaffd640 1 -- 172.21.15.6:0/1970496354 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a402cb80 con 0x7ff3ac0d3210 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.794+0000 7ff3aaffd640 1 -- 172.21.15.6:0/1970496354 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40343f0 con 0x7ff3ac0d3210 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1970496354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac0d3210 msgr2=0x7ff3ac0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1970496354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac0d3210 0x7ff3ac0d3c30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0b6820 tx=0x7ff3a402be40 comp rx=0 tx=0).stop 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1970496354 shutdown_connections 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1970496354 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac0d3210 0x7ff3ac0d3c30 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1970496354 >> v2:172.21.15.92:3300/0 conn(0x7ff3ac0ced00 0x7ff3ac0d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1970496354 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac15e020 0x7ff3ac162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1970496354 >> 172.21.15.6:0/1970496354 conn(0x7ff3ac072ce0 msgr2=0x7ff3ac0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1970496354 shutdown_connections 2024-03-18T14:44:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.795+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1970496354 wait complete. 2024-03-18T14:44:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.797+0000 7ff3b22dc640 1 Processor -- start 2024-03-18T14:44:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.797+0000 7ff3b22dc640 1 -- start start 2024-03-18T14:44:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b22dc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac0ced00 0x7ff3ac0b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b22dc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac0d3210 0x7ff3ac15c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b22dc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac15e020 0x7ff3ac152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b22dc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ac0cf440 con 0x7ff3ac15e020 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b22dc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ac0cf2c0 con 0x7ff3ac0d3210 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b22dc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff3ac0d20a0 con 0x7ff3ac0ced00 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac15e020 0x7ff3ac152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac15e020 0x7ff3ac152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37776/0 (socket says 172.21.15.6:37776) 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 -- 172.21.15.6:0/1867953748 learned_addr learned my addr 172.21.15.6:0/1867953748 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3ab7fe640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac0d3210 0x7ff3ac15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3abfff640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac0ced00 0x7ff3ac0b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 -- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac0ced00 msgr2=0x7ff3ac0b5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac0ced00 0x7ff3ac0b5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 -- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac0d3210 msgr2=0x7ff3ac15c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac0d3210 0x7ff3ac15c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.798+0000 7ff3b0879640 1 -- 172.21.15.6:0/1867953748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ac0b47d0 con 0x7ff3ac15e020 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.799+0000 7ff3b0879640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac15e020 0x7ff3ac152490 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4036040 tx=0x7ff3a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.799+0000 7ff3a8ff9640 1 -- 172.21.15.6:0/1867953748 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4043020 con 0x7ff3ac15e020 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.799+0000 7ff3a8ff9640 1 -- 172.21.15.6:0/1867953748 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a403e450 con 0x7ff3ac15e020 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.799+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3ac0b4a90 con 0x7ff3ac15e020 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.799+0000 7ff3a8ff9640 1 -- 172.21.15.6:0/1867953748 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4009510 con 0x7ff3ac15e020 2024-03-18T14:44:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.799+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3ac0b5050 con 0x7ff3ac15e020 2024-03-18T14:44:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.800+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3ac08afe0 con 0x7ff3ac15e020 2024-03-18T14:44:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.800+0000 7ff3a8ff9640 1 -- 172.21.15.6:0/1867953748 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff3a4046020 con 0x7ff3ac15e020 2024-03-18T14:44:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.801+0000 7ff3a8ff9640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff39806bbd0 0x7ff39806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.801+0000 7ff3a8ff9640 1 -- 172.21.15.6:0/1867953748 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7ff3a40ad490 con 0x7ff3ac15e020 2024-03-18T14:44:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.801+0000 7ff3abfff640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff39806bbd0 0x7ff39806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.801+0000 7ff3abfff640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff39806bbd0 0x7ff39806e090 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0006fd0 tx=0x7ff3a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:05.805+0000 7ff3a8ff9640 1 -- 172.21.15.6:0/1867953748 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff3a404d030 con 0x7ff3ac15e020 2024-03-18T14:44:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.047+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff3ac04f620 con 0x7ff3ac15e020 2024-03-18T14:44:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.048+0000 7ff3a8ff9640 1 -- 172.21.15.6:0/1867953748 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7ff3a407bdf0 con 0x7ff3ac15e020 2024-03-18T14:44:06.050 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T14:44:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff39806bbd0 msgr2=0x7ff39806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff39806bbd0 0x7ff39806e090 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0006fd0 tx=0x7ff3a0008040 comp rx=0 tx=0).stop 2024-03-18T14:44:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac15e020 msgr2=0x7ff3ac152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac15e020 0x7ff3ac152490 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4036040 tx=0x7ff3a4004a40 comp rx=0 tx=0).stop 2024-03-18T14:44:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 shutdown_connections 2024-03-18T14:44:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff39806bbd0 0x7ff39806e090 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3ac15e020 0x7ff3ac152490 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac0d3210 0x7ff3ac15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 --2- 172.21.15.6:0/1867953748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3ac0ced00 0x7ff3ac0b5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 >> 172.21.15.6:0/1867953748 conn(0x7ff3ac072ce0 msgr2=0x7ff3ac0cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 shutdown_connections 2024-03-18T14:44:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06.050+0000 7ff3b22dc640 1 -- 172.21.15.6:0/1867953748 wait complete. 2024-03-18T14:44:06.066 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T14:44:06.066 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T14:44:06.066 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-18T14:44:06.066 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T14:44:06.066 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T14:44:06.066 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T14:44:06.066 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T14:44:06.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T14:44:06.067 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:44:06.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.067 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-18T14:44:06.081 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:44:06.082 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42519sec preferred_lft 42519sec 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:44:06.083 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::c41d:adff:fe29:e32/64 scope link 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether d2:35:11:58:6e:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::d035:11ff:fe58:6e4e/64 scope link 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 92:23:d7:cf:2a:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::9023:d7ff:fecf:2a25/64 scope link 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-18T14:44:06.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.084 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T14:44:06.147 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-18T14:44:06.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.149 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-18T14:44:06.205 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-18T14:44:06.205 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T14:44:06.205 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T14:44:06.206 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T14:44:06.206 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T14:44:06.206 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T14:44:06.206 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-18T14:44:06.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T14:44:06.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T14:44:06.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T14:44:06.206 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-18T14:44:06.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T14:44:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:44:06.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.343 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:44:06.357 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T14:44:06.357 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:44:06.357 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:44:06.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-18T14:44:06.423 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T14:44:06.424 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:44:06.424 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T14:44:06.479 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T14:44:06.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T14:44:06.480 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:44:06.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:44:06.601 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:44:06.656 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:44:06.657 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:44:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:44:06.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.659 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:44:06.713 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T14:44:06.713 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T14:44:06.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-18T14:44:06.781 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-18T14:44:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T14:44:06.839 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T14:44:06.840 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-18T14:44:06.840 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:44:06.840 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:44:06.840 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:44:06.840 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:44:06.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.840 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-18T14:44:06.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:2024-03-18T14:44:06.838+0000 7f9bddc0b480 -1 init, newargv = 0x560f5fea01f0 newargc=15 2024-03-18T14:44:06.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[61864]: starting ceph client 2024-03-18T14:44:06.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[61864]: starting fuse 2024-03-18T14:44:06.855 INFO:teuthology.orchestra.run.smithi006.stdout:43 2024-03-18T14:44:06.855 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-18T14:44:06.856 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.0")) 2024-03-18T14:44:06.857 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:44:06.976 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.0.61864.asok 2024-03-18T14:44:06.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T14:44:06.986 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61864.asok 2024-03-18T14:44:06.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:06.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61864.asok status 2024-03-18T14:44:07.123 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:07.123 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-18T14:44:07.123 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "0", 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "61864", 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 6567, 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-18T14:44:07.124 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6567 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3121366599 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3121366599 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.6567 192.168.0.1:0/3121366599", 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/3121366599", 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-18T14:44:07.125 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 25, 2024-03-18T14:44:07.126 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 56, 2024-03-18T14:44:07.126 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-18T14:44:07.126 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-18T14:44:07.126 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-18T14:44:07.126 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:07.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:07.132 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-18T14:44:07.132 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T14:44:07.146 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-18T14:44:07.146 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T14:44:07.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:07.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> import errno 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> ret = os.read(fd, 1024) 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> else: 2024-03-18T14:44:07.213 DEBUG:teuthology.orchestra.run.smithi006:> os.write(fd, b'"'"'content'"'"') 2024-03-18T14:44:07.214 DEBUG:teuthology.orchestra.run.smithi006:> except OSError as e: 2024-03-18T14:44:07.214 DEBUG:teuthology.orchestra.run.smithi006:> if e.errno != errno.EPERM: 2024-03-18T14:44:07.214 DEBUG:teuthology.orchestra.run.smithi006:> raise 2024-03-18T14:44:07.214 DEBUG:teuthology.orchestra.run.smithi006:> else: 2024-03-18T14:44:07.214 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("client does not check permission of data pool") 2024-03-18T14:44:07.214 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-18T14:44:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:Traceback (most recent call last): 2024-03-18T14:44:07.319 INFO:teuthology.orchestra.run.smithi006.stderr: File "", line 15, in 2024-03-18T14:44:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:RuntimeError: client does not check permission of data pool 2024-03-18T14:44:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-18T14:44:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:07.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-18T14:44:07.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.475+0000 7f48fe590640 1 Processor -- start 2024-03-18T14:44:07.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.475+0000 7f48fe590640 1 -- start start 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48fe590640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f48f815e030 0x7f48f80d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48fe590640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 0x7f48f80d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48fe590640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f80d0e20 con 0x7f48f80b6140 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48fe590640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f48f815e410 con 0x7f48f815e030 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48fe590640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f48f80ce300 con 0x7f48f80d0450 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f48f80b6140 0x7f48f80ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45766/0 (socket says 172.21.15.6:45766) 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f77fe640 1 -- 172.21.15.6:0/305979080 learned_addr learned my addr 172.21.15.6:0/305979080 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48fcb2d640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 0x7f48f80d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f7fff640 1 --2- 172.21.15.6:0/305979080 >> v2:172.21.15.92:3300/0 conn(0x7f48f815e030 0x7f48f80d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722616603 0 0) 0x7f48f80d0e20 con 0x7f48f80b6140 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48e4003520 con 0x7f48f80b6140 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3344546909 0 0) 0x7f48e4003520 con 0x7f48f80b6140 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 msgr2=0x7f48f80d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 0x7f48f80d08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 >> v2:172.21.15.92:3300/0 conn(0x7f48f815e030 msgr2=0x7f48f80d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 --2- 172.21.15.6:0/305979080 >> v2:172.21.15.92:3300/0 conn(0x7f48f815e030 0x7f48f80d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f80b65a0 con 0x7f48f80b6140 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f48e0001670 con 0x7f48f80b6140 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f7fff640 1 --2- 172.21.15.6:0/305979080 >> v2:172.21.15.92:3300/0 conn(0x7f48f815e030 0x7f48f80d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48fcb2d640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 0x7f48f80d08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.476+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 >> v1:172.21.15.6:6789/0 conn(0x7f48f80b6140 legacy=0x7f48f80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:07.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f6ffd640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 0x7f48e40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f6ffd640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48e4003fa0 0x7f48e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f6ffd640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4004d40 0x7f48e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f48f815e410 con 0x7f48e4003fa0 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f48f80ce300 con 0x7f48e4004d40 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f48e400baa0 con 0x7f48f80d0450 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f77fe640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48e4003fa0 0x7f48e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f7fff640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 0x7f48e40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48fcb2d640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4004d40 0x7f48e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f77fe640 1 -- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 msgr2=0x7f48e40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f77fe640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f80d0450 0x7f48e40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f77fe640 1 -- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4004d40 msgr2=0x7f48e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f77fe640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4004d40 0x7f48e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.477+0000 7f48f77fe640 1 -- 172.21.15.6:0/305979080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e4003520 con 0x7f48e4003fa0 2024-03-18T14:44:07.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48f77fe640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48e4003fa0 0x7f48e4004800 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c210 tx=0x7f48e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9e6bc920f6b890 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0007d50 con 0x7f48e4003fa0 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e0030070 con 0x7f48e4003fa0 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48f6ffd640 1 -- 172.21.15.6:0/305979080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e002cdc0 con 0x7f48e4003fa0 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48fe590640 1 -- 172.21.15.6:0/305979080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48e4003fa0 msgr2=0x7f48e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48fe590640 1 --2- 172.21.15.6:0/305979080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48e4003fa0 0x7f48e4004800 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c210 tx=0x7f48e4005830 comp rx=0 tx=0).stop 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48fcb2d640 1 -- 172.21.15.6:0/305979080 reap_dead start 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48fe590640 1 -- 172.21.15.6:0/305979080 shutdown_connections 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.478+0000 7f48fe590640 1 -- 172.21.15.6:0/305979080 >> 172.21.15.6:0/305979080 conn(0x7f48f806c8c0 msgr2=0x7f48f806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.479+0000 7f48fe590640 1 -- 172.21.15.6:0/305979080 shutdown_connections 2024-03-18T14:44:07.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.479+0000 7f48fe590640 1 -- 172.21.15.6:0/305979080 wait complete. 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.480+0000 7f48fe590640 1 Processor -- start 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.480+0000 7f48fe590640 1 -- start start 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4003fa0 0x7f48f81491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f8160d10 0x7f48f8158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48f8161170 0x7f48f815e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f48f80ce550 con 0x7f48f8161170 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f48f80d3290 con 0x7f48e4003fa0 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f48f80ce830 con 0x7f48f8160d10 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48f8161170 0x7f48f815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48f8161170 0x7f48f815e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37798/0 (socket says 172.21.15.6:37798) 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 -- 172.21.15.6:0/3805956561 learned_addr learned my addr 172.21.15.6:0/3805956561 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48f77fe640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f8160d10 0x7f48f8158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48f7fff640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4003fa0 0x7f48f81491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 -- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f8160d10 msgr2=0x7f48f8158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f8160d10 0x7f48f8158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 -- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4003fa0 msgr2=0x7f48f81491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4003fa0 0x7f48f81491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 -- 172.21.15.6:0/3805956561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f8157bd0 con 0x7f48f8161170 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48f77fe640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f8160d10 0x7f48f8158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48f7fff640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4003fa0 0x7f48f81491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fcb2d640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48f8161170 0x7f48f815e590 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f48f808c380 tx=0x7f48f0010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48f4ff9640 1 -- 172.21.15.6:0/3805956561 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f0010d20 con 0x7f48f8161170 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48f4ff9640 1 -- 172.21.15.6:0/3805956561 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f0003990 con 0x7f48f8161170 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f8157eb0 con 0x7f48f8161170 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48f4ff9640 1 -- 172.21.15.6:0/3805956561 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f0003b80 con 0x7f48f8161170 2024-03-18T14:44:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.481+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f8158470 con 0x7f48f8161170 2024-03-18T14:44:07.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.482+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48f80751b0 con 0x7f48f8161170 2024-03-18T14:44:07.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.483+0000 7f48f4ff9640 1 -- 172.21.15.6:0/3805956561 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f48f001e020 con 0x7f48f8161170 2024-03-18T14:44:07.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.483+0000 7f48f4ff9640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f48d806bc00 0x7f48d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:07.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.484+0000 7f48f4ff9640 1 -- 172.21.15.6:0/3805956561 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f48f008b0c0 con 0x7f48f8161170 2024-03-18T14:44:07.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.484+0000 7f48f7fff640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f48d806bc00 0x7f48d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:07.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.484+0000 7f48f7fff640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f48d806bc00 0x7f48d806e0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f48e8006fd0 tx=0x7f48e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:07.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.488+0000 7f48f4ff9640 1 -- 172.21.15.6:0/3805956561 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f48f0025020 con 0x7f48f8161170 2024-03-18T14:44:07.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.741+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f48f815ab70 con 0x7f48f8161170 2024-03-18T14:44:07.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.744+0000 7f48f4ff9640 1 -- 172.21.15.6:0/3805956561 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v18) ==== 184+0+141 (secure 0 0 0) 0x7f48f0059ab0 con 0x7f48f8161170 2024-03-18T14:44:07.746 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-18T14:44:07.746 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBlUvhld1ffLhAAdISSYVRk06fQ2fzO68Mq0Q== 2024-03-18T14:44:07.746 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-18T14:44:07.746 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-18T14:44:07.746 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-18T14:44:07.746 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f48d806bc00 msgr2=0x7f48d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f48d806bc00 0x7f48d806e0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f48e8006fd0 tx=0x7f48e8008040 comp rx=0 tx=0).stop 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48f8161170 msgr2=0x7f48f815e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48f8161170 0x7f48f815e590 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f48f808c380 tx=0x7f48f0010530 comp rx=0 tx=0).stop 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 shutdown_connections 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f48d806bc00 0x7f48d806e0c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48f8161170 0x7f48f815e590 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48f8160d10 0x7f48f8158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 --2- 172.21.15.6:0/3805956561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48e4003fa0 0x7f48f81491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 >> 172.21.15.6:0/3805956561 conn(0x7f48f806c8c0 msgr2=0x7f48f8161a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 shutdown_connections 2024-03-18T14:44:07.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:07.746+0000 7f48fe590640 1 -- 172.21.15.6:0/3805956561 wait complete. 2024-03-18T14:44:07.759 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T14:44:07.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:44:07.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T14:44:07.853 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T14:44:07.853 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:44:07.853 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:44:07.853 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:07.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:44:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.017+0000 7f6314812640 1 Processor -- start 2024-03-18T14:44:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.017+0000 7f6314812640 1 -- start start 2024-03-18T14:44:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f6314812640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63100d0d90 0x7f63100b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f6314812640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63100b9ae0 con 0x7f63100b9650 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f6314812640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63100cef50 con 0x7f631015e170 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f6314812640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f63100cf230 con 0x7f63100d0d90 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63100b9650 0x7f63100d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45770/0 (socket says 172.21.15.6:45770) 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630dd9b640 1 -- 172.21.15.6:0/3861062935 learned_addr learned my addr 172.21.15.6:0/3861062935 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630ed9d640 1 --2- 172.21.15.6:0/3861062935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63100d0d90 0x7f63100b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3838329661 0 0) 0x7f63100b9ae0 con 0x7f63100b9650 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62fc003520 con 0x7f63100b9650 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490554567 0 0) 0x7f63100cef50 con 0x7f631015e170 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63100b9ae0 con 0x7f631015e170 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3998306909 0 0) 0x7f62fc003520 con 0x7f63100b9650 2024-03-18T14:44:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630ed9d640 1 -- 172.21.15.6:0/3861062935 >> v1:172.21.15.92:6789/0 conn(0x7f631015e170 legacy=0x7f63100d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630ed9d640 1 -- 172.21.15.6:0/3861062935 >> v1:172.21.15.6:6789/0 conn(0x7f63100b9650 legacy=0x7f63100d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.019+0000 7f630ed9d640 1 -- 172.21.15.6:0/3861062935 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63100d2b30 con 0x7f63100d0d90 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.020+0000 7f630ed9d640 1 --2- 172.21.15.6:0/3861062935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63100d0d90 0x7f63100b6dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6300009dc0 tx=0x7f630002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a1b6df755342ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.020+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300039070 con 0x7f63100d0d90 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.020+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f630002c910 con 0x7f63100d0d90 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.020+0000 7f630d59a640 1 -- 172.21.15.6:0/3861062935 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8001670 con 0x7f63100d0d90 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.021+0000 7f6314812640 1 -- 172.21.15.6:0/3861062935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63100d0d90 msgr2=0x7f63100b6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.021+0000 7f6314812640 1 --2- 172.21.15.6:0/3861062935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63100d0d90 0x7f63100b6dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6300009dc0 tx=0x7f630002bde0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.021+0000 7f6314812640 1 -- 172.21.15.6:0/3861062935 shutdown_connections 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.021+0000 7f6314812640 1 --2- 172.21.15.6:0/3861062935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63100d0d90 0x7f63100b6dd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.021+0000 7f6314812640 1 -- 172.21.15.6:0/3861062935 >> 172.21.15.6:0/3861062935 conn(0x7f631006c900 msgr2=0x7f631006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.021+0000 7f6314812640 1 -- 172.21.15.6:0/3861062935 shutdown_connections 2024-03-18T14:44:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.021+0000 7f6314812640 1 -- 172.21.15.6:0/3861062935 wait complete. 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.022+0000 7f6314812640 1 Processor -- start 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.022+0000 7f6314812640 1 -- start start 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f6314812640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63100b9650 0x7f63100ab100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f6314812640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63100d0d90 0x7f63101f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f6314812640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f631015e170 0x7f63101f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f6314812640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f63100d2820 con 0x7f63100d0d90 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f6314812640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f63100d3a40 con 0x7f63100b9650 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f6314812640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f63100ceb70 con 0x7f631015e170 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f630dd9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63100d0d90 0x7f63101f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f630dd9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63100d0d90 0x7f63101f6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37810/0 (socket says 172.21.15.6:37810) 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f630dd9b640 1 -- 172.21.15.6:0/1512509448 learned_addr learned my addr 172.21.15.6:0/1512509448 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f630ed9d640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f631015e170 0x7f63101f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.023+0000 7f630e59c640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63100b9650 0x7f63100ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630dd9b640 1 -- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f631015e170 msgr2=0x7f63101f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630dd9b640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f631015e170 0x7f63101f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630dd9b640 1 -- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63100b9650 msgr2=0x7f63100ab100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630dd9b640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63100b9650 0x7f63100ab100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630dd9b640 1 -- 172.21.15.6:0/1512509448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63101f7550 con 0x7f63100d0d90 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630ed9d640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f631015e170 0x7f63101f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630e59c640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63100b9650 0x7f63100ab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f630dd9b640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63100d0d90 0x7f63101f6e00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c780 tx=0x7f62f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f6306ffd640 1 -- 172.21.15.6:0/1512509448 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8017070 con 0x7f63100d0d90 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f6306ffd640 1 -- 172.21.15.6:0/1512509448 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f8005af0 con 0x7f63100d0d90 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63101f7830 con 0x7f63100d0d90 2024-03-18T14:44:08.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f6306ffd640 1 -- 172.21.15.6:0/1512509448 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8007070 con 0x7f63100d0d90 2024-03-18T14:44:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.024+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63101f4480 con 0x7f63100d0d90 2024-03-18T14:44:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.025+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f631008afe0 con 0x7f63100d0d90 2024-03-18T14:44:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.025+0000 7f6306ffd640 1 -- 172.21.15.6:0/1512509448 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f62f801a070 con 0x7f63100d0d90 2024-03-18T14:44:08.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.026+0000 7f6306ffd640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f62f006bbd0 0x7f62f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.026+0000 7f6306ffd640 1 -- 172.21.15.6:0/1512509448 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f62f808a0e0 con 0x7f63100d0d90 2024-03-18T14:44:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.026+0000 7f630e59c640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f62f006bbd0 0x7f62f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.027+0000 7f630e59c640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f62f006bbd0 0x7f62f006e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f63080011e0 tx=0x7f6308009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.034+0000 7f6306ffd640 1 -- 172.21.15.6:0/1512509448 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f62f8058ad0 con 0x7f63100d0d90 2024-03-18T14:44:08.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.218+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f631004f620 con 0x7f63100d0d90 2024-03-18T14:44:08.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.219+0000 7f6306ffd640 1 -- 172.21.15.6:0/1512509448 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7f62f80583d0 con 0x7f63100d0d90 2024-03-18T14:44:08.222 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:08.222 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f62f006bbd0 msgr2=0x7f62f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f62f006bbd0 0x7f62f006e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f63080011e0 tx=0x7f6308009040 comp rx=0 tx=0).stop 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63100d0d90 msgr2=0x7f63101f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63100d0d90 0x7f63101f6e00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c780 tx=0x7f62f800cc40 comp rx=0 tx=0).stop 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 shutdown_connections 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f62f006bbd0 0x7f62f006e090 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f631015e170 0x7f63101f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63100d0d90 0x7f63101f6e00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 --2- 172.21.15.6:0/1512509448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63100b9650 0x7f63100ab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 >> 172.21.15.6:0/1512509448 conn(0x7f631006c900 msgr2=0x7f63100b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 shutdown_connections 2024-03-18T14:44:08.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.221+0000 7f6314812640 1 -- 172.21.15.6:0/1512509448 wait complete. 2024-03-18T14:44:08.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:43:39.597879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:43:40.299491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:44:08.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:44:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.338+0000 7f226b2f1640 1 Processor -- start 2024-03-18T14:44:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.338+0000 7f226b2f1640 1 -- start start 2024-03-18T14:44:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226b2f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f226415df80 0x7f22640d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226b2f1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22640cea40 0x7f22640ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226b2f1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22640d2540 con 0x7f22640b98b0 2024-03-18T14:44:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226b2f1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f22640d2820 con 0x7f22640cea40 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226b2f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f22640d2b00 con 0x7f226415df80 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226988e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22640cea40 0x7f22640ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226988e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22640cea40 0x7f22640ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37826/0 (socket says 172.21.15.6:37826) 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226988e640 1 -- 172.21.15.6:0/2747817452 learned_addr learned my addr 172.21.15.6:0/2747817452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226908d640 1 --2- 172.21.15.6:0/2747817452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f226415df80 0x7f22640d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226988e640 1 -- 172.21.15.6:0/2747817452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f226415df80 msgr2=0x7f22640d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226988e640 1 --2- 172.21.15.6:0/2747817452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f226415df80 0x7f22640d35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226988e640 1 -- 172.21.15.6:0/2747817452 >> v1:172.21.15.92:6789/0 conn(0x7f22640b98b0 legacy=0x7f22640d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.339+0000 7f226988e640 1 -- 172.21.15.6:0/2747817452 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22640d2de0 con 0x7f22640cea40 2024-03-18T14:44:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.340+0000 7f226988e640 1 --2- 172.21.15.6:0/2747817452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f22640ceed0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f225c009e50 tx=0x7f225c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05643b81d9a83d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.340+0000 7f225bfff640 1 -- 172.21.15.6:0/2747817452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c02cb00 con 0x7f22640cea40 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.340+0000 7f225bfff640 1 -- 172.21.15.6:0/2747817452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225c02cca0 con 0x7f22640cea40 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.340+0000 7f225bfff640 1 -- 172.21.15.6:0/2747817452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c034610 con 0x7f22640cea40 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 -- 172.21.15.6:0/2747817452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 msgr2=0x7f22640ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 --2- 172.21.15.6:0/2747817452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f22640ceed0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f225c009e50 tx=0x7f225c02bf60 comp rx=0 tx=0).stop 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 -- 172.21.15.6:0/2747817452 shutdown_connections 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 --2- 172.21.15.6:0/2747817452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f22640ceed0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 --2- 172.21.15.6:0/2747817452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f226415df80 0x7f22640d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 -- 172.21.15.6:0/2747817452 >> 172.21.15.6:0/2747817452 conn(0x7f2264072d40 msgr2=0x7f2264073150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 -- 172.21.15.6:0/2747817452 shutdown_connections 2024-03-18T14:44:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.341+0000 7f226b2f1640 1 -- 172.21.15.6:0/2747817452 wait complete. 2024-03-18T14:44:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.342+0000 7f226b2f1640 1 Processor -- start 2024-03-18T14:44:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.342+0000 7f226b2f1640 1 -- start start 2024-03-18T14:44:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226b2f1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22640b98b0 0x7f2264155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226b2f1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f2264157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226b2f1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f226415df80 0x7f22641579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226b2f1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f22640b6da0 con 0x7f22640cea40 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226b2f1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f226415e360 con 0x7f226415df80 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226b2f1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f22640b6f20 con 0x7f22640b98b0 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f2264157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f2264157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37840/0 (socket says 172.21.15.6:37840) 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 -- 172.21.15.6:0/4164650202 learned_addr learned my addr 172.21.15.6:0/4164650202 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226908d640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22640b98b0 0x7f2264155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226988e640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f226415df80 0x7f22641579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 -- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22640b98b0 msgr2=0x7f2264155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22640b98b0 0x7f2264155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 -- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f226415df80 msgr2=0x7f22641579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f226415df80 0x7f22641579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 -- 172.21.15.6:0/4164650202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226415eaf0 con 0x7f22640cea40 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226988e640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f226415df80 0x7f22641579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226908d640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22640b98b0 0x7f2264155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226888c640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f2264157490 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f225400c400 tx=0x7f225400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f2259ffb640 1 -- 172.21.15.6:0/4164650202 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22540052d0 con 0x7f22640cea40 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f2259ffb640 1 -- 172.21.15.6:0/4164650202 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2254005490 con 0x7f22640cea40 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f226415edd0 con 0x7f22640cea40 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.343+0000 7f2259ffb640 1 -- 172.21.15.6:0/4164650202 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254007260 con 0x7f22640cea40 2024-03-18T14:44:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.344+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f226415f390 con 0x7f22640cea40 2024-03-18T14:44:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.345+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22640751b0 con 0x7f22640cea40 2024-03-18T14:44:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.345+0000 7f2259ffb640 1 -- 172.21.15.6:0/4164650202 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2254007400 con 0x7f22640cea40 2024-03-18T14:44:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.345+0000 7f2259ffb640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f224806bbd0 0x7f224806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.346+0000 7f2259ffb640 1 -- 172.21.15.6:0/4164650202 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f225408a090 con 0x7f22640cea40 2024-03-18T14:44:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.346+0000 7f226908d640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f224806bbd0 0x7f224806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.346+0000 7f226908d640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f224806bbd0 0x7f224806e090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2260006fd0 tx=0x7f2260008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.350+0000 7f2259ffb640 1 -- 172.21.15.6:0/4164650202 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2254058a80 con 0x7f22640cea40 2024-03-18T14:44:08.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.535+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f22640d0fa0 con 0x7f22640cea40 2024-03-18T14:44:08.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.536+0000 7f2259ffb640 1 -- 172.21.15.6:0/4164650202 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f2254058380 con 0x7f22640cea40 2024-03-18T14:44:08.538 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:44:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f224806bbd0 msgr2=0x7f224806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f224806bbd0 0x7f224806e090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2260006fd0 tx=0x7f2260008040 comp rx=0 tx=0).stop 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 msgr2=0x7f2264157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f2264157490 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f225400c400 tx=0x7f225400c8c0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 shutdown_connections 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f224806bbd0 0x7f224806e090 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f226415df80 0x7f22641579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22640cea40 0x7f2264157490 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 --2- 172.21.15.6:0/4164650202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22640b98b0 0x7f2264155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 >> 172.21.15.6:0/4164650202 conn(0x7f2264072d40 msgr2=0x7f22640b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 shutdown_connections 2024-03-18T14:44:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.538+0000 7f226b2f1640 1 -- 172.21.15.6:0/4164650202 wait complete. 2024-03-18T14:44:08.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:44:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.656+0000 7f5374dcd640 1 Processor -- start 2024-03-18T14:44:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.656+0000 7f5374dcd640 1 -- start start 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f5374dcd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f5374dcd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f53700d0450 0x7f53700d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f5374dcd640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53700d0e20 con 0x7f53700b6140 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f5374dcd640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f537015e410 con 0x7f53700d0450 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f5374dcd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f53700ce300 con 0x7f537015e030 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f536ed9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f53700d0450 0x7f53700d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f536ed9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f53700d0450 0x7f53700d08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37846/0 (socket says 172.21.15.6:37846) 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.657+0000 7f536ed9d640 1 -- 172.21.15.6:0/3799866712 learned_addr learned my addr 172.21.15.6:0/3799866712 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.658+0000 7f536e59c640 1 --2- 172.21.15.6:0/3799866712 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.658+0000 7f536ed9d640 1 -- 172.21.15.6:0/3799866712 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 msgr2=0x7f53700d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.658+0000 7f536ed9d640 1 --2- 172.21.15.6:0/3799866712 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.658+0000 7f536ed9d640 1 -- 172.21.15.6:0/3799866712 >> v1:172.21.15.92:6789/0 conn(0x7f53700b6140 legacy=0x7f53700ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.658+0000 7f536ed9d640 1 -- 172.21.15.6:0/3799866712 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53700b65a0 con 0x7f53700d0450 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.658+0000 7f536ed9d640 1 --2- 172.21.15.6:0/3799866712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700d08e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5360009e50 tx=0x7f536002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712fd248306f5d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f536d59a640 1 -- 172.21.15.6:0/3799866712 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536002cb00 con 0x7f53700d0450 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f536d59a640 1 -- 172.21.15.6:0/3799866712 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f536002cca0 con 0x7f53700d0450 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f536d59a640 1 -- 172.21.15.6:0/3799866712 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5360034610 con 0x7f53700d0450 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 -- 172.21.15.6:0/3799866712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 msgr2=0x7f53700d08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3799866712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700d08e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5360009e50 tx=0x7f536002bf60 comp rx=0 tx=0).stop 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 -- 172.21.15.6:0/3799866712 shutdown_connections 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3799866712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700d08e0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3799866712 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 -- 172.21.15.6:0/3799866712 >> 172.21.15.6:0/3799866712 conn(0x7f537006c8c0 msgr2=0x7f537006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 -- 172.21.15.6:0/3799866712 shutdown_connections 2024-03-18T14:44:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.659+0000 7f5374dcd640 1 -- 172.21.15.6:0/3799866712 wait complete. 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 Processor -- start 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 -- start start 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53700b6140 0x7f5370155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700b8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f53700ce550 con 0x7f53700d0450 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53700d3290 con 0x7f53700b6140 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f5374dcd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f53700ce830 con 0x7f537015e030 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f536dd9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f536dd9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37854/0 (socket says 172.21.15.6:37854) 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.661+0000 7f536dd9b640 1 -- 172.21.15.6:0/3640402015 learned_addr learned my addr 172.21.15.6:0/3640402015 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536ed9d640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700b8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536e59c640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53700b6140 0x7f5370155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536dd9b640 1 -- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 msgr2=0x7f53700b8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536dd9b640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700b8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536dd9b640 1 -- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53700b6140 msgr2=0x7f5370155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536dd9b640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53700b6140 0x7f5370155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536dd9b640 1 -- 172.21.15.6:0/3640402015 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53701e7c00 con 0x7f53700d0450 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536e59c640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53700b6140 0x7f5370155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f536dd9b640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700cc990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f536400c400 tx=0x7f536400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f535effd640 1 -- 172.21.15.6:0/3640402015 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53640052d0 con 0x7f53700d0450 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f535effd640 1 -- 172.21.15.6:0/3640402015 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5364005490 con 0x7f53700d0450 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53701e7ee0 con 0x7f53700d0450 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f535effd640 1 -- 172.21.15.6:0/3640402015 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5364007260 con 0x7f53700d0450 2024-03-18T14:44:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.662+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53701e8780 con 0x7f53700d0450 2024-03-18T14:44:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.663+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53700751b0 con 0x7f53700d0450 2024-03-18T14:44:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.664+0000 7f535effd640 1 -- 172.21.15.6:0/3640402015 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5364007400 con 0x7f53700d0450 2024-03-18T14:44:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.664+0000 7f535effd640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f535006bbd0 0x7f535006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.664+0000 7f535effd640 1 -- 172.21.15.6:0/3640402015 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f536408a4f0 con 0x7f53700d0450 2024-03-18T14:44:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.665+0000 7f536e59c640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f535006bbd0 0x7f535006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.665+0000 7f536e59c640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f535006bbd0 0x7f535006e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5368006fd0 tx=0x7f5368008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.668+0000 7f535effd640 1 -- 172.21.15.6:0/3640402015 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f53640588e0 con 0x7f53700d0450 2024-03-18T14:44:08.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.849+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f53700d21b0 con 0x7f53700d0450 2024-03-18T14:44:08.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.849+0000 7f535effd640 1 -- 172.21.15.6:0/3640402015 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f536405c980 con 0x7f53700d0450 2024-03-18T14:44:08.854 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f535006bbd0 msgr2=0x7f535006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f535006bbd0 0x7f535006e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5368006fd0 tx=0x7f5368008040 comp rx=0 tx=0).stop 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 msgr2=0x7f53700cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700cc990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f536400c400 tx=0x7f536400c8c0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 shutdown_connections 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f535006bbd0 0x7f535006e090 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f537015e030 0x7f53700b8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53700d0450 0x7f53700cc990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 --2- 172.21.15.6:0/3640402015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53700b6140 0x7f5370155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 >> 172.21.15.6:0/3640402015 conn(0x7f537006c8c0 msgr2=0x7f53700b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 shutdown_connections 2024-03-18T14:44:08.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.851+0000 7f5374dcd640 1 -- 172.21.15.6:0/3640402015 wait complete. 2024-03-18T14:44:08.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:44:08.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.970+0000 7fc696b62640 1 Processor -- start 2024-03-18T14:44:08.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.970+0000 7fc696b62640 1 -- start start 2024-03-18T14:44:08.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.970+0000 7fc696b62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900d0730 0x7fc6900cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.970+0000 7fc696b62640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6900d2510 0x7fc69015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.970+0000 7fc696b62640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6900d2960 con 0x7fc6900ce9d0 2024-03-18T14:44:08.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.970+0000 7fc696b62640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc6900cd8d0 con 0x7fc6900d2510 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.970+0000 7fc696b62640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc6900cfac0 con 0x7fc6900d0730 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc6950ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6900d2510 0x7fc69015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc6948fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6900ce9d0 0x7fc6900d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45778/0 (socket says 172.21.15.6:45778) 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc6948fe640 1 -- 172.21.15.6:0/93443637 learned_addr learned my addr 172.21.15.6:0/93443637 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc68ffff640 1 --2- 172.21.15.6:0/93443637 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900d0730 0x7fc6900cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc6950ff640 1 -- 172.21.15.6:0/93443637 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900d0730 msgr2=0x7fc6900cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc6950ff640 1 --2- 172.21.15.6:0/93443637 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900d0730 0x7fc6900cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc6950ff640 1 -- 172.21.15.6:0/93443637 >> v1:172.21.15.6:6789/0 conn(0x7fc6900ce9d0 legacy=0x7fc6900d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc6950ff640 1 -- 172.21.15.6:0/93443637 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6900cccc0 con 0x7fc6900d2510 2024-03-18T14:44:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.971+0000 7fc68f7fe640 1 -- 172.21.15.6:0/93443637 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 433542958 0 0) 0x7fc6900d2960 con 0x7fc6900ce9d0 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc6950ff640 1 --2- 172.21.15.6:0/93443637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc69015b690 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc688009d90 tx=0x7fc68802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d09367dfbc4f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc68f7fe640 1 -- 172.21.15.6:0/93443637 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688039070 con 0x7fc6900d2510 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc68f7fe640 1 -- 172.21.15.6:0/93443637 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc68802cbc0 con 0x7fc6900d2510 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc68f7fe640 1 -- 172.21.15.6:0/93443637 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688034440 con 0x7fc6900d2510 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 -- 172.21.15.6:0/93443637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 msgr2=0x7fc69015b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 --2- 172.21.15.6:0/93443637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc69015b690 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc688009d90 tx=0x7fc68802be10 comp rx=0 tx=0).stop 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 -- 172.21.15.6:0/93443637 shutdown_connections 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 --2- 172.21.15.6:0/93443637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc69015b690 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 --2- 172.21.15.6:0/93443637 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900d0730 0x7fc6900cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 -- 172.21.15.6:0/93443637 >> 172.21.15.6:0/93443637 conn(0x7fc69008b0c0 msgr2=0x7fc6900d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 -- 172.21.15.6:0/93443637 shutdown_connections 2024-03-18T14:44:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.972+0000 7fc696b62640 1 -- 172.21.15.6:0/93443637 wait complete. 2024-03-18T14:44:08.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.973+0000 7fc696b62640 1 Processor -- start 2024-03-18T14:44:08.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.973+0000 7fc696b62640 1 -- start start 2024-03-18T14:44:08.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc696b62640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900ce9d0 0x7fc6901f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc696b62640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6900d0730 0x7fc690156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc696b62640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc6900cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc696b62640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc6900cd560 con 0x7fc6900d2510 2024-03-18T14:44:08.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc696b62640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc6900cd280 con 0x7fc6900d0730 2024-03-18T14:44:08.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc696b62640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc6900cedb0 con 0x7fc6900ce9d0 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc6950ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc6900cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc6950ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc6900cb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37872/0 (socket says 172.21.15.6:37872) 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.974+0000 7fc6950ff640 1 -- 172.21.15.6:0/3526195705 learned_addr learned my addr 172.21.15.6:0/3526195705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc68ffff640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6900d0730 0x7fc690156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6948fe640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900ce9d0 0x7fc6901f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6950ff640 1 -- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900ce9d0 msgr2=0x7fc6901f4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6950ff640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900ce9d0 0x7fc6901f4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6950ff640 1 -- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6900d0730 msgr2=0x7fc690156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6950ff640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6900d0730 0x7fc690156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6950ff640 1 -- 172.21.15.6:0/3526195705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6900b5de0 con 0x7fc6900d2510 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6948fe640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900ce9d0 0x7fc6901f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc6950ff640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc6900cb330 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc688009d90 tx=0x7fc688004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc68d7fa640 1 -- 172.21.15.6:0/3526195705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688039050 con 0x7fc6900d2510 2024-03-18T14:44:08.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc68d7fa640 1 -- 172.21.15.6:0/3526195705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc688045070 con 0x7fc6900d2510 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6900b6080 con 0x7fc6900d2510 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc68d7fa640 1 -- 172.21.15.6:0/3526195705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688009450 con 0x7fc6900d2510 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.975+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6900b6640 con 0x7fc6900d2510 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.976+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6900751b0 con 0x7fc6900d2510 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.977+0000 7fc68d7fa640 1 -- 172.21.15.6:0/3526195705 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc68803e850 con 0x7fc6900d2510 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.977+0000 7fc68d7fa640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc67c06bbd0 0x7fc67c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.977+0000 7fc68d7fa640 1 -- 172.21.15.6:0/3526195705 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fc6880adb00 con 0x7fc6900d2510 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.977+0000 7fc6948fe640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc67c06bbd0 0x7fc67c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.978+0000 7fc6948fe640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc67c06bbd0 0x7fc67c06e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc6800010e0 tx=0x7fc680009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:08.984+0000 7fc68d7fa640 1 -- 172.21.15.6:0/3526195705 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc68807c580 con 0x7fc6900d2510 2024-03-18T14:44:09.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.167+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc6900d1f30 con 0x7fc6900d2510 2024-03-18T14:44:09.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.167+0000 7fc68d7fa640 1 -- 172.21.15.6:0/3526195705 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7fc68807be80 con 0x7fc6900d2510 2024-03-18T14:44:09.170 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc67c06bbd0 msgr2=0x7fc67c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc67c06bbd0 0x7fc67c06e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc6800010e0 tx=0x7fc680009040 comp rx=0 tx=0).stop 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 msgr2=0x7fc6900cb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc6900cb330 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc688009d90 tx=0x7fc688004a40 comp rx=0 tx=0).stop 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 shutdown_connections 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc67c06bbd0 0x7fc67c06e090 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6900d2510 0x7fc6900cb330 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6900d0730 0x7fc690156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 --2- 172.21.15.6:0/3526195705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc6900ce9d0 0x7fc6901f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 >> 172.21.15.6:0/3526195705 conn(0x7fc69008b0c0 msgr2=0x7fc69015e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 shutdown_connections 2024-03-18T14:44:09.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.169+0000 7fc696b62640 1 -- 172.21.15.6:0/3526195705 wait complete. 2024-03-18T14:44:09.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T14:44:09.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.283+0000 7fe1005e6640 1 Processor -- start 2024-03-18T14:44:09.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.283+0000 7fe1005e6640 1 -- start start 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe1005e6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0f806c7a0 0x7fe0f80d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe1005e6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe0f806cb80 0x7fe0f80b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe1005e6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0f80d1f30 con 0x7fe0f80b6df0 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe1005e6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe0f80d2210 con 0x7fe0f806cb80 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe1005e6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe0f80d24f0 con 0x7fe0f806c7a0 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe0fdb81640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe0f806cb80 0x7fe0f80b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe0feb83640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe0f80b6df0 0x7fe0f80b7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45780/0 (socket says 172.21.15.6:45780) 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.284+0000 7fe0feb83640 1 -- 172.21.15.6:0/738632938 learned_addr learned my addr 172.21.15.6:0/738632938 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.285+0000 7fe0fe382640 1 --2- 172.21.15.6:0/738632938 >> v2:172.21.15.92:3300/0 conn(0x7fe0f806c7a0 0x7fe0f80d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.285+0000 7fe0fdb81640 1 -- 172.21.15.6:0/738632938 >> v2:172.21.15.92:3300/0 conn(0x7fe0f806c7a0 msgr2=0x7fe0f80d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.285+0000 7fe0fdb81640 1 --2- 172.21.15.6:0/738632938 >> v2:172.21.15.92:3300/0 conn(0x7fe0f806c7a0 0x7fe0f80d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.285+0000 7fe0fdb81640 1 -- 172.21.15.6:0/738632938 >> v1:172.21.15.6:6789/0 conn(0x7fe0f80b6df0 legacy=0x7fe0f80b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.285+0000 7fe0fdb81640 1 -- 172.21.15.6:0/738632938 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f80d27d0 con 0x7fe0f806cb80 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.285+0000 7fe0fd380640 1 -- 172.21.15.6:0/738632938 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053434079 0 0) 0x7fe0f80d1f30 con 0x7fe0f80b6df0 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.285+0000 7fe0fe382640 1 --2- 172.21.15.6:0/738632938 >> v2:172.21.15.92:3300/0 conn(0x7fe0f806c7a0 0x7fe0f80d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe0fdb81640 1 --2- 172.21.15.6:0/738632938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f80b95e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80d0fa0 tx=0x7fe0e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58efcfda6cb0467c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe0fd380640 1 -- 172.21.15.6:0/738632938 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8034070 con 0x7fe0f806cb80 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe0fd380640 1 -- 172.21.15.6:0/738632938 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0e8030610 con 0x7fe0f806cb80 2024-03-18T14:44:09.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe0fd380640 1 -- 172.21.15.6:0/738632938 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e80309e0 con 0x7fe0f806cb80 2024-03-18T14:44:09.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe1005e6640 1 -- 172.21.15.6:0/738632938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 msgr2=0x7fe0f80b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe1005e6640 1 --2- 172.21.15.6:0/738632938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f80b95e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80d0fa0 tx=0x7fe0e802b960 comp rx=0 tx=0).stop 2024-03-18T14:44:09.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe1005e6640 1 -- 172.21.15.6:0/738632938 shutdown_connections 2024-03-18T14:44:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe1005e6640 1 --2- 172.21.15.6:0/738632938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f80b95e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe1005e6640 1 --2- 172.21.15.6:0/738632938 >> v2:172.21.15.92:3300/0 conn(0x7fe0f806c7a0 0x7fe0f80d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.286+0000 7fe1005e6640 1 -- 172.21.15.6:0/738632938 >> 172.21.15.6:0/738632938 conn(0x7fe0f8072ce0 msgr2=0x7fe0f80730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.287+0000 7fe1005e6640 1 -- 172.21.15.6:0/738632938 shutdown_connections 2024-03-18T14:44:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.287+0000 7fe1005e6640 1 -- 172.21.15.6:0/738632938 wait complete. 2024-03-18T14:44:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.288+0000 7fe1005e6640 1 Processor -- start 2024-03-18T14:44:09.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.288+0000 7fe1005e6640 1 -- start start 2024-03-18T14:44:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.289+0000 7fe1005e6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0f806c7a0 0x7fe0f81f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe1005e6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f81542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe1005e6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0f80b6df0 0x7fe0f80b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe1005e6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f80b9b20 con 0x7fe0f806cb80 2024-03-18T14:44:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe1005e6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f806cfe0 con 0x7fe0f80b6df0 2024-03-18T14:44:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe1005e6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe0f80b9ca0 con 0x7fe0f806c7a0 2024-03-18T14:44:09.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f81542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f81542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37898/0 (socket says 172.21.15.6:37898) 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 -- 172.21.15.6:0/598041591 learned_addr learned my addr 172.21.15.6:0/598041591 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0feb83640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0f80b6df0 0x7fe0f80b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fe382640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0f806c7a0 0x7fe0f81f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 -- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0f806c7a0 msgr2=0x7fe0f81f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0f806c7a0 0x7fe0f81f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 -- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0f80b6df0 msgr2=0x7fe0f80b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0f80b6df0 0x7fe0f80b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 -- 172.21.15.6:0/598041591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f815e450 con 0x7fe0f806cb80 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fe382640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0f806c7a0 0x7fe0f81f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0fdb81640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f81542d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe0e802bf70 tx=0x7fe0e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0eeffd640 1 -- 172.21.15.6:0/598041591 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8034040 con 0x7fe0f806cb80 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0eeffd640 1 -- 172.21.15.6:0/598041591 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0e8039a10 con 0x7fe0f806cb80 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe0eeffd640 1 -- 172.21.15.6:0/598041591 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8039de0 con 0x7fe0f806cb80 2024-03-18T14:44:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.290+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f81f7a00 con 0x7fe0f806cb80 2024-03-18T14:44:09.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.291+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f815f800 con 0x7fe0f806cb80 2024-03-18T14:44:09.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.291+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0f804f5b0 con 0x7fe0f806cb80 2024-03-18T14:44:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.292+0000 7fe0eeffd640 1 -- 172.21.15.6:0/598041591 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe0e8037070 con 0x7fe0f806cb80 2024-03-18T14:44:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.292+0000 7fe0eeffd640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0dc06bbd0 0x7fe0dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.293+0000 7fe0eeffd640 1 -- 172.21.15.6:0/598041591 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fe0e80accc0 con 0x7fe0f806cb80 2024-03-18T14:44:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.293+0000 7fe0fe382640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0dc06bbd0 0x7fe0dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.293+0000 7fe0fe382640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0dc06bbd0 0x7fe0dc06e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400aa30 tx=0x7fe0f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.297+0000 7fe0eeffd640 1 -- 172.21.15.6:0/598041591 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe0e807b740 con 0x7fe0f806cb80 2024-03-18T14:44:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.493+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe0f80cd7b0 con 0x7fe0f806cb80 2024-03-18T14:44:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.493+0000 7fe0eeffd640 1 -- 172.21.15.6:0/598041591 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7fe0e807b040 con 0x7fe0f806cb80 2024-03-18T14:44:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0dc06bbd0 msgr2=0x7fe0dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0dc06bbd0 0x7fe0dc06e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400aa30 tx=0x7fe0f4008040 comp rx=0 tx=0).stop 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 msgr2=0x7fe0f81542d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f81542d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe0e802bf70 tx=0x7fe0e80046c0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 shutdown_connections 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe0dc06bbd0 0x7fe0dc06e090 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0f80b6df0 0x7fe0f80b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0f806cb80 0x7fe0f81542d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 --2- 172.21.15.6:0/598041591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0f806c7a0 0x7fe0f81f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 >> 172.21.15.6:0/598041591 conn(0x7fe0f8072ce0 msgr2=0x7fe0f80ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 shutdown_connections 2024-03-18T14:44:09.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.496+0000 7fe1005e6640 1 -- 172.21.15.6:0/598041591 wait complete. 2024-03-18T14:44:09.512 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-18T14:44:09.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:09.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T14:44:09.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi006.stderr:ceph-fuse[61864]: fuse finished with error 0 and tester_r 0 2024-03-18T14:44:09.539 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:09.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:09.540 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T14:44:09.632 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:44:09.633 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:44:09.633 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:09.651 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:44:09.651 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:44:09.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:09.651 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T14:44:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:09.708 INFO:teuthology.orchestra.run.smithi092.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T14:44:09.708 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:44:09.708 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:44:09.708 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:09.723 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2314 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-18T14:44:09.723 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:09.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:09.723 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T14:44:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:09.777 INFO:teuthology.orchestra.run.smithi006.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T14:44:09.777 INFO:tasks.cephfs.mount:Terminating background process 2024-03-18T14:44:09.777 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:44:09.777 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:09.795 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:44:09.795 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:44:09.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:09.795 DEBUG:teuthology.orchestra.run.smithi092:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T14:44:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:09.852 INFO:teuthology.orchestra.run.smithi092.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T14:44:09.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:44:09.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.954+0000 7fcb43a9a640 1 Processor -- start 2024-03-18T14:44:09.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.954+0000 7fcb43a9a640 1 -- start start 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb43a9a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15ad40 0x7fcb3c06b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb43a9a640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb3c06b700 0x7fcb3c15ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb43a9a640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb3c15bc30 0x7fcb3c15f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb43a9a640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fcb3c15c1d0 con 0x7fcb3c06b700 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb43a9a640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcb3c15e8d0 con 0x7fcb3c15bc30 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb43a9a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcb3c0b31a0 con 0x7fcb3c15ad40 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb3c06b700 0x7fcb3c15ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb3c06b700 0x7fcb3c15ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37906/0 (socket says 172.21.15.6:37906) 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 -- 172.21.15.6:0/3385657668 learned_addr learned my addr 172.21.15.6:0/3385657668 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41836640 1 --2- 172.21.15.6:0/3385657668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15ad40 0x7fcb3c06b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb42037640 1 --2- 172.21.15.6:0/3385657668 >> v2:172.21.15.92:3300/0 conn(0x7fcb3c15bc30 0x7fcb3c15f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 -- 172.21.15.6:0/3385657668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15ad40 msgr2=0x7fcb3c06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:09.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 --2- 172.21.15.6:0/3385657668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15ad40 0x7fcb3c06b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 -- 172.21.15.6:0/3385657668 >> v2:172.21.15.92:3300/0 conn(0x7fcb3c15bc30 msgr2=0x7fcb3c15f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:09.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 --2- 172.21.15.6:0/3385657668 >> v2:172.21.15.92:3300/0 conn(0x7fcb3c15bc30 0x7fcb3c15f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41035640 1 -- 172.21.15.6:0/3385657668 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3c0b3440 con 0x7fcb3c06b700 2024-03-18T14:44:09.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb41836640 1 --2- 172.21.15.6:0/3385657668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15ad40 0x7fcb3c06b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:09.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.956+0000 7fcb42037640 1 --2- 172.21.15.6:0/3385657668 >> v2:172.21.15.92:3300/0 conn(0x7fcb3c15bc30 0x7fcb3c15f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.957+0000 7fcb41035640 1 --2- 172.21.15.6:0/3385657668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c06b700 0x7fcb3c15ed60 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0ae1d0 tx=0x7fcb2c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b533376e962e587f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.957+0000 7fcb40834640 1 -- 172.21.15.6:0/3385657668 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c03b070 con 0x7fcb3c06b700 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.957+0000 7fcb40834640 1 -- 172.21.15.6:0/3385657668 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb2c036440 con 0x7fcb3c06b700 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.957+0000 7fcb40834640 1 -- 172.21.15.6:0/3385657668 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c036810 con 0x7fcb3c06b700 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 -- 172.21.15.6:0/3385657668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c06b700 msgr2=0x7fcb3c15ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/3385657668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c06b700 0x7fcb3c15ed60 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0ae1d0 tx=0x7fcb2c02e0e0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 -- 172.21.15.6:0/3385657668 shutdown_connections 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/3385657668 >> v2:172.21.15.92:3300/0 conn(0x7fcb3c15bc30 0x7fcb3c15f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/3385657668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c06b700 0x7fcb3c15ed60 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/3385657668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15ad40 0x7fcb3c06b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 -- 172.21.15.6:0/3385657668 >> 172.21.15.6:0/3385657668 conn(0x7fcb3c0ccc30 msgr2=0x7fcb3c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 -- 172.21.15.6:0/3385657668 shutdown_connections 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.958+0000 7fcb43a9a640 1 -- 172.21.15.6:0/3385657668 wait complete. 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.959+0000 7fcb43a9a640 1 Processor -- start 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.959+0000 7fcb43a9a640 1 -- start start 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb43a9a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb3c06b700 0x7fcb3c1f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb43a9a640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c15ad40 0x7fcb3c0c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb43a9a640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15bc30 0x7fcb3c0c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb43a9a640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c0734f0 con 0x7fcb3c15ad40 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb43a9a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c073370 con 0x7fcb3c06b700 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb43a9a640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcb3c072220 con 0x7fcb3c15bc30 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c15ad40 0x7fcb3c0c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c15ad40 0x7fcb3c0c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37914/0 (socket says 172.21.15.6:37914) 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 -- 172.21.15.6:0/1583006844 learned_addr learned my addr 172.21.15.6:0/1583006844 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41836640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb3c06b700 0x7fcb3c1f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb42037640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15bc30 0x7fcb3c0c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 -- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15bc30 msgr2=0x7fcb3c0c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15bc30 0x7fcb3c0c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 -- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb3c06b700 msgr2=0x7fcb3c1f70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb3c06b700 0x7fcb3c1f70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 -- 172.21.15.6:0/1583006844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3c0cde00 con 0x7fcb3c15ad40 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb41035640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c15ad40 0x7fcb3c0c9840 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c02e120 tx=0x7fcb2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.960+0000 7fcb327fc640 1 -- 172.21.15.6:0/1583006844 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c03b030 con 0x7fcb3c15ad40 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.961+0000 7fcb327fc640 1 -- 172.21.15.6:0/1583006844 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb2c03d070 con 0x7fcb3c15ad40 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.961+0000 7fcb327fc640 1 -- 172.21.15.6:0/1583006844 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c034bc0 con 0x7fcb3c15ad40 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.961+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb3c0ce0e0 con 0x7fcb3c15ad40 2024-03-18T14:44:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.961+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb3c0ce6a0 con 0x7fcb3c15ad40 2024-03-18T14:44:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.962+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb3c09e260 con 0x7fcb3c15ad40 2024-03-18T14:44:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.962+0000 7fcb327fc640 1 -- 172.21.15.6:0/1583006844 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcb2c03f030 con 0x7fcb3c15ad40 2024-03-18T14:44:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.962+0000 7fcb327fc640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcb2006bbd0 0x7fcb2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.962+0000 7fcb327fc640 1 -- 172.21.15.6:0/1583006844 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fcb2c0ac8b0 con 0x7fcb3c15ad40 2024-03-18T14:44:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.962+0000 7fcb41836640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcb2006bbd0 0x7fcb2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.963+0000 7fcb41836640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcb2006bbd0 0x7fcb2006e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0c6f10 tx=0x7fcb38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:09.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:09.967+0000 7fcb327fc640 1 -- 172.21.15.6:0/1583006844 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcb2c07ac50 con 0x7fcb3c15ad40 2024-03-18T14:44:10.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.188+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb3c15d710 con 0x7fcb3c15ad40 2024-03-18T14:44:10.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.188+0000 7fcb327fc640 1 -- 172.21.15.6:0/1583006844 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7fcb2c07ecf0 con 0x7fcb3c15ad40 2024-03-18T14:44:10.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:10.191 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/1499402578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1499402578},{"type":"v1","addr":"172.21.15.92:6836","nonce":1499402578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6022,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/300544617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":300544617},{"type":"v1","addr":"172.21.15.6:6835","nonce":300544617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/175151750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":175151750},{"type":"v1","addr":"172.21.15.92:6837","nonce":175151750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:43:41.338763+0000","modified":"2024-03-18T14:43:44.344988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6225},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6225":{"gid":6225,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/3070832840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3070832840},{"type":"v1","addr":"172.21.15.6:6834","nonce":3070832840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6225,"qdb_cluster":[6225]},"id":3}]} 2024-03-18T14:44:10.191 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcb2006bbd0 msgr2=0x7fcb2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcb2006bbd0 0x7fcb2006e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0c6f10 tx=0x7fcb38008040 comp rx=0 tx=0).stop 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c15ad40 msgr2=0x7fcb3c0c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c15ad40 0x7fcb3c0c9840 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c02e120 tx=0x7fcb2c0046c0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 shutdown_connections 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcb2006bbd0 0x7fcb2006e090 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb3c15bc30 0x7fcb3c0c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb3c15ad40 0x7fcb3c0c9840 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 --2- 172.21.15.6:0/1583006844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb3c06b700 0x7fcb3c1f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 >> 172.21.15.6:0/1583006844 conn(0x7fcb3c0ccc30 msgr2=0x7fcb3c15c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:10.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 shutdown_connections 2024-03-18T14:44:10.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.190+0000 7fcb43a9a640 1 -- 172.21.15.6:0/1583006844 wait complete. 2024-03-18T14:44:10.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:44:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.303+0000 7f0297b87640 1 Processor -- start 2024-03-18T14:44:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.303+0000 7f0297b87640 1 -- start start 2024-03-18T14:44:10.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0297b87640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f029006d3f0 0x7f02900cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0297b87640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900b28b0 con 0x7f02900b1b50 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0297b87640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900b2b90 con 0x7f029015def0 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0297b87640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f02900d0540 con 0x7f029006d3f0 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0295122640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f029006d3f0 0x7f02900cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0295122640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f029006d3f0 0x7f02900cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37916/0 (socket says 172.21.15.6:37916) 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0295122640 1 -- 172.21.15.6:0/2784753990 learned_addr learned my addr 172.21.15.6:0/2784753990 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0295122640 1 -- 172.21.15.6:0/2784753990 >> v1:172.21.15.92:6789/0 conn(0x7f029015def0 legacy=0x7f029006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0295122640 1 -- 172.21.15.6:0/2784753990 >> v1:172.21.15.6:6789/0 conn(0x7f02900b1b50 legacy=0x7f02900b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0295122640 1 -- 172.21.15.6:0/2784753990 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02900d01d0 con 0x7f029006d3f0 2024-03-18T14:44:10.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.304+0000 7f0294921640 1 -- 172.21.15.6:0/2784753990 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155556464 0 0) 0x7f02900b28b0 con 0x7f02900b1b50 2024-03-18T14:44:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.305+0000 7f0295122640 1 --2- 172.21.15.6:0/2784753990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029006d3f0 0x7f02900cc1a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f02900d06c0 tx=0x7f028002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbaa9f2d0ec9de4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.305+0000 7f0294921640 1 -- 172.21.15.6:0/2784753990 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028002f840 con 0x7f029006d3f0 2024-03-18T14:44:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.305+0000 7f0294921640 1 -- 172.21.15.6:0/2784753990 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f028002f9e0 con 0x7f029006d3f0 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.305+0000 7f0294921640 1 -- 172.21.15.6:0/2784753990 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028002fdb0 con 0x7f029006d3f0 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.306+0000 7f0297b87640 1 -- 172.21.15.6:0/2784753990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029006d3f0 msgr2=0x7f02900cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.306+0000 7f0297b87640 1 --2- 172.21.15.6:0/2784753990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029006d3f0 0x7f02900cc1a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f02900d06c0 tx=0x7f028002ba20 comp rx=0 tx=0).stop 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.306+0000 7f0297b87640 1 -- 172.21.15.6:0/2784753990 shutdown_connections 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.306+0000 7f0297b87640 1 --2- 172.21.15.6:0/2784753990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029006d3f0 0x7f02900cc1a0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.306+0000 7f0297b87640 1 -- 172.21.15.6:0/2784753990 >> 172.21.15.6:0/2784753990 conn(0x7f029009ae20 msgr2=0x7f029008b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.306+0000 7f0297b87640 1 -- 172.21.15.6:0/2784753990 shutdown_connections 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.306+0000 7f0297b87640 1 -- 172.21.15.6:0/2784753990 wait complete. 2024-03-18T14:44:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.307+0000 7f0297b87640 1 Processor -- start 2024-03-18T14:44:10.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.307+0000 7f0297b87640 1 -- start start 2024-03-18T14:44:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0297b87640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029006d3f0 0x7f02900b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0297b87640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02900b1b50 0x7f02900c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0297b87640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029015def0 0x7f0290159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0297b87640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f02900d09d0 con 0x7f029015def0 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0297b87640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f029015e2d0 con 0x7f02900b1b50 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0297b87640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f02900d0b50 con 0x7f029006d3f0 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029015def0 0x7f0290159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029015def0 0x7f0290159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37932/0 (socket says 172.21.15.6:37932) 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 -- 172.21.15.6:0/3437132863 learned_addr learned my addr 172.21.15.6:0/3437132863 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0295923640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029006d3f0 0x7f02900b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0295122640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02900b1b50 0x7f02900c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 -- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029006d3f0 msgr2=0x7f02900b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029006d3f0 0x7f02900b0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 -- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02900b1b50 msgr2=0x7f02900c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02900b1b50 0x7f02900c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 -- 172.21.15.6:0/3437132863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02900c8dc0 con 0x7f029015def0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.308+0000 7f0296124640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029015def0 0x7f0290159780 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f028800c760 tx=0x7f028800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.309+0000 7f02867fc640 1 -- 172.21.15.6:0/3437132863 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288003da0 con 0x7f029015def0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.309+0000 7f02867fc640 1 -- 172.21.15.6:0/3437132863 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0288005070 con 0x7f029015def0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.309+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02900c90a0 con 0x7f029015def0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.309+0000 7f02867fc640 1 -- 172.21.15.6:0/3437132863 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288005460 con 0x7f029015def0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.309+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02900c9660 con 0x7f029015def0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.310+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f029006fd60 con 0x7f029015def0 2024-03-18T14:44:10.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.310+0000 7f02867fc640 1 -- 172.21.15.6:0/3437132863 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0288005600 con 0x7f029015def0 2024-03-18T14:44:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.311+0000 7f02867fc640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f027406bbd0 0x7f027406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.311+0000 7f02867fc640 1 -- 172.21.15.6:0/3437132863 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f028808ac10 con 0x7f029015def0 2024-03-18T14:44:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.311+0000 7f0295923640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f027406bbd0 0x7f027406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.312+0000 7f0295923640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f027406bbd0 0x7f027406e090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f028c000f60 tx=0x7f028c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.315+0000 7f02867fc640 1 -- 172.21.15.6:0/3437132863 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f029006fd60 con 0x7f029015def0 2024-03-18T14:44:10.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.504+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f029009fa90 con 0x7f029015def0 2024-03-18T14:44:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.506+0000 7f02867fc640 1 -- 172.21.15.6:0/3437132863 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7f02880594e0 con 0x7f029015def0 2024-03-18T14:44:10.508 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:10.508 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:44:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f027406bbd0 msgr2=0x7f027406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f027406bbd0 0x7f027406e090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f028c000f60 tx=0x7f028c009040 comp rx=0 tx=0).stop 2024-03-18T14:44:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029015def0 msgr2=0x7f0290159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029015def0 0x7f0290159780 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f028800c760 tx=0x7f028800cc20 comp rx=0 tx=0).stop 2024-03-18T14:44:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 shutdown_connections 2024-03-18T14:44:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f027406bbd0 0x7f027406e090 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f029015def0 0x7f0290159780 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02900b1b50 0x7f02900c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.507+0000 7f0297b87640 1 --2- 172.21.15.6:0/3437132863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029006d3f0 0x7f02900b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.508+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 >> 172.21.15.6:0/3437132863 conn(0x7f029009ae20 msgr2=0x7f02900d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:10.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.508+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 shutdown_connections 2024-03-18T14:44:10.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.508+0000 7f0297b87640 1 -- 172.21.15.6:0/3437132863 wait complete. 2024-03-18T14:44:10.521 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T14:41:23.427896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T14:43:39.597879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T14:43:40.299491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T14:44:10.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T14:44:10.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.624+0000 7fd303d13640 1 Processor -- start 2024-03-18T14:44:10.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.624+0000 7fd303d13640 1 -- start start 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd303d13640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd2fc0d0b60 0x7fd2fc0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd303d13640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2fc0d39d0 con 0x7fd2fc0d3570 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd303d13640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2fc0d2fd0 con 0x7fd2fc15e030 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd303d13640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd2fc15e410 con 0x7fd2fc0d0b60 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd3012ae640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd2fc0d3570 0x7fd2fc0d0450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45796/0 (socket says 172.21.15.6:45796) 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd3012ae640 1 -- 172.21.15.6:0/2113852789 learned_addr learned my addr 172.21.15.6:0/2113852789 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd3022b0640 1 --2- 172.21.15.6:0/2113852789 >> v2:172.21.15.92:3300/0 conn(0x7fd2fc0d0b60 0x7fd2fc0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 398260204 0 0) 0x7fd2fc0d39d0 con 0x7fd2fc0d3570 2024-03-18T14:44:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2e8003520 con 0x7fd2fc0d3570 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.625+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2563226962 0 0) 0x7fd2e8003520 con 0x7fd2fc0d3570 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 >> v2:172.21.15.92:3300/0 conn(0x7fd2fc0d0b60 msgr2=0x7fd2fc0b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 --2- 172.21.15.6:0/2113852789 >> v2:172.21.15.92:3300/0 conn(0x7fd2fc0d0b60 0x7fd2fc0b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 >> v1:172.21.15.92:6789/0 conn(0x7fd2fc15e030 legacy=0x7fd2fc0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc0d3290 con 0x7fd2fc0d3570 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fd2ec001670 con 0x7fd2fc0d3570 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 >> v1:172.21.15.6:6789/0 conn(0x7fd2fc0d3570 legacy=0x7fd2fc0d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2e80039b0 0x7fd2e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2e8004800 0x7fd2e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e80051c0 0x7fd2e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc15e410 con 0x7fd2e8004800 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e800c060 con 0x7fd2e80039b0 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e800c260 con 0x7fd2e80051c0 2024-03-18T14:44:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3012ae640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2e8004800 0x7fd2e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd301aaf640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2e80039b0 0x7fd2e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3022b0640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e80051c0 0x7fd2e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3012ae640 1 -- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e80051c0 msgr2=0x7fd2e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3012ae640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e80051c0 0x7fd2e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3012ae640 1 -- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2e80039b0 msgr2=0x7fd2e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3012ae640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2e80039b0 0x7fd2e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3012ae640 1 -- 172.21.15.6:0/2113852789 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e8003520 con 0x7fd2e8004800 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.626+0000 7fd3022b0640 1 -- 172.21.15.6:0/2113852789 reap_dead start 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.627+0000 7fd3012ae640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2e8004800 0x7fd2e8004c80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c210 tx=0x7fd2e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34ee0c251d396e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.627+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec007d30 con 0x7fd2e8004800 2024-03-18T14:44:10.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.627+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ec030070 con 0x7fd2e8004800 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.627+0000 7fd300aad640 1 -- 172.21.15.6:0/2113852789 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec005d70 con 0x7fd2e8004800 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.627+0000 7fd303d13640 1 -- 172.21.15.6:0/2113852789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2e8004800 msgr2=0x7fd2e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.627+0000 7fd303d13640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2e8004800 0x7fd2e8004c80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c210 tx=0x7fd2e8005830 comp rx=0 tx=0).stop 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.628+0000 7fd303d13640 1 -- 172.21.15.6:0/2113852789 shutdown_connections 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.628+0000 7fd303d13640 1 --2- 172.21.15.6:0/2113852789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2e8004800 0x7fd2e8004c80 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.628+0000 7fd303d13640 1 -- 172.21.15.6:0/2113852789 >> 172.21.15.6:0/2113852789 conn(0x7fd2fc06c8c0 msgr2=0x7fd2fc06ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.628+0000 7fd303d13640 1 -- 172.21.15.6:0/2113852789 shutdown_connections 2024-03-18T14:44:10.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.628+0000 7fd303d13640 1 -- 172.21.15.6:0/2113852789 wait complete. 2024-03-18T14:44:10.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.629+0000 7fd303d13640 1 Processor -- start 2024-03-18T14:44:10.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.629+0000 7fd303d13640 1 -- start start 2024-03-18T14:44:10.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd303d13640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e8004800 0x7fd2fc149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd303d13640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2fc14a1e0 0x7fd2fc1f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd303d13640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2fc1f79f0 0x7fd2fc0cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd303d13640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc0ce830 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd303d13640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc0ce300 con 0x7fd2fc14a1e0 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd303d13640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd2fc0b6140 con 0x7fd2e8004800 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2fc1f79f0 0x7fd2fc0cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2fc1f79f0 0x7fd2fc0cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37954/0 (socket says 172.21.15.6:37954) 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 -- 172.21.15.6:0/3088937535 learned_addr learned my addr 172.21.15.6:0/3088937535 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd301aaf640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e8004800 0x7fd2fc149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3012ae640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2fc14a1e0 0x7fd2fc1f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 -- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e8004800 msgr2=0x7fd2fc149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e8004800 0x7fd2fc149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 -- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2fc14a1e0 msgr2=0x7fd2fc1f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2fc14a1e0 0x7fd2fc1f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3022b0640 1 -- 172.21.15.6:0/3088937535 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc157490 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.630+0000 7fd3012ae640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2fc14a1e0 0x7fd2fc1f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.631+0000 7fd3022b0640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2fc1f79f0 0x7fd2fc0cb900 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0cfb00 tx=0x7fd2f4011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.631+0000 7fd2f27fc640 1 -- 172.21.15.6:0/3088937535 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4019020 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.631+0000 7fd2f27fc640 1 -- 172.21.15.6:0/3088937535 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f4004030 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.631+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc0b48c0 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.631+0000 7fd2f27fc640 1 -- 172.21.15.6:0/3088937535 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4004400 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.631+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2fc0cae20 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.632+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2fc0751b0 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.632+0000 7fd2f27fc640 1 -- 172.21.15.6:0/3088937535 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd2f401c020 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.633+0000 7fd2f27fc640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd2e006bbd0 0x7fd2e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.633+0000 7fd2f27fc640 1 -- 172.21.15.6:0/3088937535 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fd2f408ae20 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.633+0000 7fd301aaf640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd2e006bbd0 0x7fd2e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.633+0000 7fd301aaf640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd2e006bbd0 0x7fd2e006e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8000f80 tx=0x7fd2f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.637+0000 7fd2f27fc640 1 -- 172.21.15.6:0/3088937535 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd2f4023020 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.820+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2fc0d19f0 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.821+0000 7fd2f27fc640 1 -- 172.21.15.6:0/3088937535 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fd2f4059810 con 0x7fd2fc1f79f0 2024-03-18T14:44:10.823 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-18T14:44:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.822+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd2e006bbd0 msgr2=0x7fd2e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.822+0000 7fd303d13640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd2e006bbd0 0x7fd2e006e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8000f80 tx=0x7fd2f8009040 comp rx=0 tx=0).stop 2024-03-18T14:44:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.822+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2fc1f79f0 msgr2=0x7fd2fc0cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.822+0000 7fd303d13640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2fc1f79f0 0x7fd2fc0cb900 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0cfb00 tx=0x7fd2f4011cf0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 shutdown_connections 2024-03-18T14:44:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd2e006bbd0 0x7fd2e006e090 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2fc1f79f0 0x7fd2fc0cb900 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2fc14a1e0 0x7fd2fc1f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 --2- 172.21.15.6:0/3088937535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2e8004800 0x7fd2fc149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 >> 172.21.15.6:0/3088937535 conn(0x7fd2fc06c8c0 msgr2=0x7fd2fc15d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 shutdown_connections 2024-03-18T14:44:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.823+0000 7fd303d13640 1 -- 172.21.15.6:0/3088937535 wait complete. 2024-03-18T14:44:10.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T14:44:10.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.939+0000 7f7326ef1640 1 Processor -- start 2024-03-18T14:44:10.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.939+0000 7f7326ef1640 1 -- start start 2024-03-18T14:44:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7326ef1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f732015e150 0x7f73200d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7326ef1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7326ef1640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73200d0300 con 0x7f732006d400 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7326ef1640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f732015e530 con 0x7f732015e150 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7326ef1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f73200d05e0 con 0x7f732009a350 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7324c8d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f732015e150 0x7f73200d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f732548e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7324c8d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f732015e150 0x7f73200d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51956/0 (socket says 172.21.15.6:51956) 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7324c8d640 1 -- 172.21.15.6:0/2488131721 learned_addr learned my addr 172.21.15.6:0/2488131721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7324c8d640 1 -- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 msgr2=0x7f73200cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7324c8d640 1 --2- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7324c8d640 1 -- 172.21.15.6:0/2488131721 >> v1:172.21.15.92:6789/0 conn(0x7f732006d400 legacy=0x7f7320099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f7324c8d640 1 -- 172.21.15.6:0/2488131721 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73200d08c0 con 0x7f732015e150 2024-03-18T14:44:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.940+0000 7f732548e640 1 --2- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.941+0000 7f7324c8d640 1 --2- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732015e150 0x7f73200d13a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7314009870 tx=0x7f731402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44ac839aa1376733 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.941+0000 7f731f7fe640 1 -- 172.21.15.6:0/2488131721 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7314035020 con 0x7f732015e150 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.941+0000 7f731f7fe640 1 -- 172.21.15.6:0/2488131721 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7314031640 con 0x7f732015e150 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.941+0000 7f731f7fe640 1 -- 172.21.15.6:0/2488131721 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731402d040 con 0x7f732015e150 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 -- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732015e150 msgr2=0x7f73200d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 --2- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732015e150 0x7f73200d13a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7314009870 tx=0x7f731402b8f0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 -- 172.21.15.6:0/2488131721 shutdown_connections 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 --2- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 --2- 172.21.15.6:0/2488131721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732015e150 0x7f73200d13a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 -- 172.21.15.6:0/2488131721 >> 172.21.15.6:0/2488131721 conn(0x7f73200b3650 msgr2=0x7f73200b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 -- 172.21.15.6:0/2488131721 shutdown_connections 2024-03-18T14:44:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.942+0000 7f7326ef1640 1 -- 172.21.15.6:0/2488131721 wait complete. 2024-03-18T14:44:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.944+0000 7f7326ef1640 1 Processor -- start 2024-03-18T14:44:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.944+0000 7f7326ef1640 1 -- start start 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.945+0000 7f7326ef1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732006d400 0x7f7320158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.945+0000 7f7326ef1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.945+0000 7f7326ef1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f732015e150 0x7f73200c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.945+0000 7f7326ef1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f732006d890 con 0x7f732015e150 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.945+0000 7f7326ef1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f73200d0e80 con 0x7f732006d400 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.945+0000 7f7326ef1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f732006c650 con 0x7f732009a350 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.945+0000 7f732548e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f732015e150 0x7f73200c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f732015e150 0x7f73200c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37962/0 (socket says 172.21.15.6:37962) 2024-03-18T14:44:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 -- 172.21.15.6:0/378505994 learned_addr learned my addr 172.21.15.6:0/378505994 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f731ffff640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f7324c8d640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732006d400 0x7f7320158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 -- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 msgr2=0x7f73200c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 -- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732006d400 msgr2=0x7f7320158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732006d400 0x7f7320158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 -- 172.21.15.6:0/378505994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732015f0c0 con 0x7f732015e150 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f7324c8d640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732006d400 0x7f7320158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f732548e640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f732015e150 0x7f73200c9170 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f731800efc0 tx=0x7f7318009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f731d7fa640 1 -- 172.21.15.6:0/378505994 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7318018020 con 0x7f732015e150 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f731d7fa640 1 -- 172.21.15.6:0/378505994 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7318014480 con 0x7f732015e150 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f731d7fa640 1 -- 172.21.15.6:0/378505994 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7318014870 con 0x7f732015e150 2024-03-18T14:44:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732015f360 con 0x7f732015e150 2024-03-18T14:44:10.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.946+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f732015f920 con 0x7f732015e150 2024-03-18T14:44:10.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.947+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f732015ffe0 con 0x7f732015e150 2024-03-18T14:44:10.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.948+0000 7f731d7fa640 1 -- 172.21.15.6:0/378505994 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7318005d10 con 0x7f732015e150 2024-03-18T14:44:10.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.948+0000 7f731d7fa640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f730406bbd0 0x7f730406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:10.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.948+0000 7f731d7fa640 1 -- 172.21.15.6:0/378505994 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f731808b9c0 con 0x7f732015e150 2024-03-18T14:44:10.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.949+0000 7f7324c8d640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f730406bbd0 0x7f730406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:10.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.949+0000 7f7324c8d640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f730406bbd0 0x7f730406e090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7314036000 tx=0x7f731403c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:10.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:10.953+0000 7f731d7fa640 1 -- 172.21.15.6:0/378505994 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f732015ffe0 con 0x7f732015e150 2024-03-18T14:44:11.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.166+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f732006e3c0 con 0x7f732015e150 2024-03-18T14:44:11.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.166+0000 7f731d7fa640 1 -- 172.21.15.6:0/378505994 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f732006e3c0 con 0x7f732015e150 2024-03-18T14:44:11.168 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f730406bbd0 msgr2=0x7f730406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f730406bbd0 0x7f730406e090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7314036000 tx=0x7f731403c040 comp rx=0 tx=0).stop 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f732015e150 msgr2=0x7f73200c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f732015e150 0x7f73200c9170 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f731800efc0 tx=0x7f7318009f90 comp rx=0 tx=0).stop 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 shutdown_connections 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f730406bbd0 0x7f730406e090 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f732015e150 0x7f73200c9170 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f732009a350 0x7f73200c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 --2- 172.21.15.6:0/378505994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f732006d400 0x7f7320158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 >> 172.21.15.6:0/378505994 conn(0x7f73200b3650 msgr2=0x7f73200b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 shutdown_connections 2024-03-18T14:44:11.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.168+0000 7f7326ef1640 1 -- 172.21.15.6:0/378505994 wait complete. 2024-03-18T14:44:11.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T14:44:11.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.287+0000 7f708a940640 1 Processor -- start 2024-03-18T14:44:11.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.287+0000 7f708a940640 1 -- start start 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f708a940640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f708a940640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70840d0680 0x7f708415d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f708a940640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70840d0f30 con 0x7f70840ce930 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f708a940640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f70840b1070 con 0x7f70840d0680 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f708a940640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f70840b1350 con 0x7f708406d1f0 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7088edd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70840d0680 0x7f708415d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7083fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70840ce930 0x7f708406cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45812/0 (socket says 172.21.15.6:45812) 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7083fff640 1 -- 172.21.15.6:0/1798583008 learned_addr learned my addr 172.21.15.6:0/1798583008 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f70837fe640 1 --2- 172.21.15.6:0/1798583008 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7088edd640 1 -- 172.21.15.6:0/1798583008 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 msgr2=0x7f70840d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:11.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7088edd640 1 --2- 172.21.15.6:0/1798583008 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7088edd640 1 -- 172.21.15.6:0/1798583008 >> v1:172.21.15.6:6789/0 conn(0x7f70840ce930 legacy=0x7f708406cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7088edd640 1 -- 172.21.15.6:0/1798583008 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70840d0b10 con 0x7f70840d0680 2024-03-18T14:44:11.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f7082ffd640 1 -- 172.21.15.6:0/1798583008 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573940134 0 0) 0x7f70840d0f30 con 0x7f70840ce930 2024-03-18T14:44:11.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.288+0000 7f70837fe640 1 --2- 172.21.15.6:0/1798583008 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:11.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.289+0000 7f7088edd640 1 --2- 172.21.15.6:0/1798583008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f708415d6b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f707c009d90 tx=0x7f707c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77493c64f546650 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.289+0000 7f7082ffd640 1 -- 172.21.15.6:0/1798583008 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c039070 con 0x7f70840d0680 2024-03-18T14:44:11.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.289+0000 7f7082ffd640 1 -- 172.21.15.6:0/1798583008 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707c02cc50 con 0x7f70840d0680 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.289+0000 7f7082ffd640 1 -- 172.21.15.6:0/1798583008 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c034440 con 0x7f70840d0680 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 -- 172.21.15.6:0/1798583008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 msgr2=0x7f708415d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 --2- 172.21.15.6:0/1798583008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f708415d6b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f707c009d90 tx=0x7f707c02bea0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 -- 172.21.15.6:0/1798583008 shutdown_connections 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 --2- 172.21.15.6:0/1798583008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f708415d6b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 --2- 172.21.15.6:0/1798583008 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 -- 172.21.15.6:0/1798583008 >> 172.21.15.6:0/1798583008 conn(0x7f708408b0c0 msgr2=0x7f70840d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 -- 172.21.15.6:0/1798583008 shutdown_connections 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.290+0000 7f708a940640 1 -- 172.21.15.6:0/1798583008 wait complete. 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.291+0000 7f708a940640 1 Processor -- start 2024-03-18T14:44:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.291+0000 7f708a940640 1 -- start start 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f708a940640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f708a940640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70840ce930 0x7f70840c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f708a940640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f70840af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f708a940640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f708415dbf0 con 0x7f70840d0680 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f708a940640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70840cef20 con 0x7f70840ce930 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f708a940640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f708415dd70 con 0x7f708406d1f0 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f70840af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f70840af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37988/0 (socket says 172.21.15.6:37988) 2024-03-18T14:44:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 -- 172.21.15.6:0/3086193627 learned_addr learned my addr 172.21.15.6:0/3086193627 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f70837fe640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70840ce930 0x7f70840c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7083fff640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 -- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 msgr2=0x7f70840b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 -- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70840ce930 msgr2=0x7f70840c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70840ce930 0x7f70840c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7088edd640 1 -- 172.21.15.6:0/3086193627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70840c7b60 con 0x7f70840d0680 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f70837fe640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70840ce930 0x7f70840c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.292+0000 7f7083fff640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.293+0000 7f7088edd640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f70840af8d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f707c009d90 tx=0x7f707c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.293+0000 7f7080ff9640 1 -- 172.21.15.6:0/3086193627 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c044020 con 0x7f70840d0680 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.293+0000 7f7080ff9640 1 -- 172.21.15.6:0/3086193627 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707c02ca50 con 0x7f70840d0680 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.293+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70840c7e40 con 0x7f70840d0680 2024-03-18T14:44:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.293+0000 7f7080ff9640 1 -- 172.21.15.6:0/3086193627 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c039040 con 0x7f70840d0680 2024-03-18T14:44:11.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.293+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70840c7850 con 0x7f70840d0680 2024-03-18T14:44:11.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.294+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70840751b0 con 0x7f70840d0680 2024-03-18T14:44:11.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.294+0000 7f7080ff9640 1 -- 172.21.15.6:0/3086193627 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f707c047020 con 0x7f70840d0680 2024-03-18T14:44:11.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.295+0000 7f7080ff9640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707006bbd0 0x7f707006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.295+0000 7f7080ff9640 1 -- 172.21.15.6:0/3086193627 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f707c0ad6d0 con 0x7f70840d0680 2024-03-18T14:44:11.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.295+0000 7f7083fff640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707006bbd0 0x7f707006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.296+0000 7f7083fff640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707006bbd0 0x7f707006e090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f70740010e0 tx=0x7f7074009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.299+0000 7f7080ff9640 1 -- 172.21.15.6:0/3086193627 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f707c04e030 con 0x7f70840d0680 2024-03-18T14:44:11.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.480+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f70840d01f0 con 0x7f70840d0680 2024-03-18T14:44:11.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.480+0000 7f7080ff9640 1 -- 172.21.15.6:0/3086193627 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f707c02cc20 con 0x7f70840d0680 2024-03-18T14:44:11.483 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707006bbd0 msgr2=0x7f707006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707006bbd0 0x7f707006e090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f70740010e0 tx=0x7f7074009040 comp rx=0 tx=0).stop 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 msgr2=0x7f70840af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f70840af8d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f707c009d90 tx=0x7f707c004a40 comp rx=0 tx=0).stop 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 shutdown_connections 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f707006bbd0 0x7f707006e090 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70840d0680 0x7f70840af8d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70840ce930 0x7f70840c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 --2- 172.21.15.6:0/3086193627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f708406d1f0 0x7f70840b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.482+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 >> 172.21.15.6:0/3086193627 conn(0x7f708408b0c0 msgr2=0x7f708415d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.483+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 shutdown_connections 2024-03-18T14:44:11.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.483+0000 7f708a940640 1 -- 172.21.15.6:0/3086193627 wait complete. 2024-03-18T14:44:11.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:44:11.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.629+0000 7fe3b53e6640 1 Processor -- start 2024-03-18T14:44:11.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.629+0000 7fe3b53e6640 1 -- start start 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3b53e6640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3b00ccc20 0x7fe3b00afb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3b53e6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3b53e6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3b006bbd0 0x7fe3b00cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3b53e6640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe3b00cb6b0 con 0x7fe3b00ccc20 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3b53e6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fe3b00cab00 con 0x7fe3b006bbd0 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3b53e6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b00ad630 con 0x7fe3b00ad1d0 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3b00ccc20 0x7fe3b00afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3b00ccc20 0x7fe3b00afb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38004/0 (socket says 172.21.15.6:38004) 2024-03-18T14:44:11.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 -- 172.21.15.6:0/1151120617 learned_addr learned my addr 172.21.15.6:0/1151120617 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3af7fe640 1 --2- 172.21.15.6:0/1151120617 >> v2:172.21.15.92:3300/0 conn(0x7fe3b006bbd0 0x7fe3b00cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3ae7fc640 1 --2- 172.21.15.6:0/1151120617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 -- 172.21.15.6:0/1151120617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 msgr2=0x7fe3b00cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 --2- 172.21.15.6:0/1151120617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 -- 172.21.15.6:0/1151120617 >> v2:172.21.15.92:3300/0 conn(0x7fe3b006bbd0 msgr2=0x7fe3b00cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 --2- 172.21.15.6:0/1151120617 >> v2:172.21.15.92:3300/0 conn(0x7fe3b006bbd0 0x7fe3b00cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3aeffd640 1 -- 172.21.15.6:0/1151120617 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b00cd540 con 0x7fe3b00ccc20 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.630+0000 7fe3af7fe640 1 --2- 172.21.15.6:0/1151120617 >> v2:172.21.15.92:3300/0 conn(0x7fe3b006bbd0 0x7fe3b00cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.631+0000 7fe3aeffd640 1 --2- 172.21.15.6:0/1151120617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00afb00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0067f50 tx=0x7fe3a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f47c273c34869b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.631+0000 7fe3adffb640 1 -- 172.21.15.6:0/1151120617 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a002e070 con 0x7fe3b00ccc20 2024-03-18T14:44:11.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.631+0000 7fe3adffb640 1 -- 172.21.15.6:0/1151120617 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a0037070 con 0x7fe3b00ccc20 2024-03-18T14:44:11.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.631+0000 7fe3adffb640 1 -- 172.21.15.6:0/1151120617 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0031a00 con 0x7fe3b00ccc20 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 -- 172.21.15.6:0/1151120617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b00ccc20 msgr2=0x7fe3b00afb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/1151120617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00afb00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0067f50 tx=0x7fe3a002b880 comp rx=0 tx=0).stop 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 -- 172.21.15.6:0/1151120617 shutdown_connections 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/1151120617 >> v2:172.21.15.92:3300/0 conn(0x7fe3b006bbd0 0x7fe3b00cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/1151120617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/1151120617 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00afb00 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 -- 172.21.15.6:0/1151120617 >> 172.21.15.6:0/1151120617 conn(0x7fe3b00b5340 msgr2=0x7fe3b00b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 -- 172.21.15.6:0/1151120617 shutdown_connections 2024-03-18T14:44:11.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.632+0000 7fe3b53e6640 1 -- 172.21.15.6:0/1151120617 wait complete. 2024-03-18T14:44:11.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.634+0000 7fe3b53e6640 1 Processor -- start 2024-03-18T14:44:11.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.634+0000 7fe3b53e6640 1 -- start start 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3b53e6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b006bbd0 0x7fe3b00af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3b53e6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3b53e6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3b53e6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b00cb340 con 0x7fe3b006bbd0 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3b53e6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b00cb060 con 0x7fe3b00ccc20 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3b53e6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b015c1f0 con 0x7fe3b00ad1d0 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3aeffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b006bbd0 0x7fe3b00af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3aeffd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b006bbd0 0x7fe3b00af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38010/0 (socket says 172.21.15.6:38010) 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3aeffd640 1 -- 172.21.15.6:0/4017429216 learned_addr learned my addr 172.21.15.6:0/4017429216 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3af7fe640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3ae7fc640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3aeffd640 1 -- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 msgr2=0x7fe3b00aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.635+0000 7fe3aeffd640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3aeffd640 1 -- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b00ccc20 msgr2=0x7fe3b00c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3aeffd640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3aeffd640 1 -- 172.21.15.6:0/4017429216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b00c61c0 con 0x7fe3b006bbd0 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3af7fe640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3ae7fc640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3aeffd640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b006bbd0 0x7fe3b00af810 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000c00 tx=0x7fe3a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3977fe640 1 -- 172.21.15.6:0/4017429216 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0032880 con 0x7fe3b006bbd0 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3977fe640 1 -- 172.21.15.6:0/4017429216 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a0032a20 con 0x7fe3b006bbd0 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b00c64a0 con 0x7fe3b006bbd0 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3977fe640 1 -- 172.21.15.6:0/4017429216 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0032e10 con 0x7fe3b006bbd0 2024-03-18T14:44:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.636+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b00c6a60 con 0x7fe3b006bbd0 2024-03-18T14:44:11.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.637+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3b009e550 con 0x7fe3b006bbd0 2024-03-18T14:44:11.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.637+0000 7fe3977fe640 1 -- 172.21.15.6:0/4017429216 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe3a004c060 con 0x7fe3b006bbd0 2024-03-18T14:44:11.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.638+0000 7fe3977fe640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe39c06bbd0 0x7fe39c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.638+0000 7fe3977fe640 1 -- 172.21.15.6:0/4017429216 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fe3a00ac850 con 0x7fe3b006bbd0 2024-03-18T14:44:11.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.638+0000 7fe3ae7fc640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe39c06bbd0 0x7fe39c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.638+0000 7fe3ae7fc640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe39c06bbd0 0x7fe39c06e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00682a0 tx=0x7fe3a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.642+0000 7fe3977fe640 1 -- 172.21.15.6:0/4017429216 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe3a0040070 con 0x7fe3b006bbd0 2024-03-18T14:44:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.861+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3b015c580 con 0x7fe3b006bbd0 2024-03-18T14:44:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.861+0000 7fe3977fe640 1 -- 172.21.15.6:0/4017429216 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7fe3a007b1b0 con 0x7fe3b006bbd0 2024-03-18T14:44:11.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:11.864 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/1499402578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1499402578},{"type":"v1","addr":"172.21.15.92:6836","nonce":1499402578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6022,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/300544617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":300544617},{"type":"v1","addr":"172.21.15.6:6835","nonce":300544617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/175151750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":175151750},{"type":"v1","addr":"172.21.15.92:6837","nonce":175151750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:43:41.338763+0000","modified":"2024-03-18T14:43:44.344988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6225},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6225":{"gid":6225,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/3070832840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3070832840},{"type":"v1","addr":"172.21.15.6:6834","nonce":3070832840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6225,"qdb_cluster":[6225]},"id":3}]} 2024-03-18T14:44:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe39c06bbd0 msgr2=0x7fe39c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe39c06bbd0 0x7fe39c06e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00682a0 tx=0x7fe3a4008040 comp rx=0 tx=0).stop 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b006bbd0 msgr2=0x7fe3b00af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b006bbd0 0x7fe3b00af810 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000c00 tx=0x7fe3a00046c0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 shutdown_connections 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fe39c06bbd0 0x7fe39c06e090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b00ccc20 0x7fe3b00c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b00ad1d0 0x7fe3b00aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 --2- 172.21.15.6:0/4017429216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3b006bbd0 0x7fe3b00af810 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 >> 172.21.15.6:0/4017429216 conn(0x7fe3b00b5340 msgr2=0x7fe3b00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 shutdown_connections 2024-03-18T14:44:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.863+0000 7fe3b53e6640 1 -- 172.21.15.6:0/4017429216 wait complete. 2024-03-18T14:44:11.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:44:11.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.976+0000 7fa73f344640 1 Processor -- start 2024-03-18T14:44:11.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.976+0000 7fa73f344640 1 -- start start 2024-03-18T14:44:11.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.977+0000 7fa73f344640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 0x7fa7380ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.977+0000 7fa73f344640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7380b6560 con 0x7fa738161a30 2024-03-18T14:44:11.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.977+0000 7fa73f344640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7380d2100 con 0x7fa7380d35f0 2024-03-18T14:44:11.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.977+0000 7fa73f344640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa73815e3b0 con 0x7fa73815dfd0 2024-03-18T14:44:11.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.977+0000 7fa73c8df640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa738161a30 0x7fa7380b5740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45822/0 (socket says 172.21.15.6:45822) 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.977+0000 7fa73c8df640 1 -- 172.21.15.6:0/4007924926 learned_addr learned my addr 172.21.15.6:0/4007924926 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.977+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3557187450 0 0) 0x7fa7380b6560 con 0x7fa738161a30 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa724003520 con 0x7fa738161a30 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 0x7fa7380ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 755293873 0 0) 0x7fa724003520 con 0x7fa738161a30 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 msgr2=0x7fa7380ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 0x7fa7380ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 >> v1:172.21.15.92:6789/0 conn(0x7fa7380d35f0 legacy=0x7fa7380b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa738161f60 con 0x7fa738161a30 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7fa734001670 con 0x7fa738161a30 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 >> v1:172.21.15.6:6789/0 conn(0x7fa738161a30 legacy=0x7fa7380b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7240039e0 0x7fa724003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 0x7fa724004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724004d40 0x7fa7240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa73815e3b0 con 0x7fa7240039e0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa72400ba70 con 0x7fa724004d40 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa72400bc70 con 0x7fa73815dfd0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7240039e0 0x7fa724003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73c8df640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 0x7fa724004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d8e1640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724004d40 0x7fa7240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d0e0640 1 -- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 msgr2=0x7fa724004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73815dfd0 0x7fa724004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d0e0640 1 -- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724004d40 msgr2=0x7fa7240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724004d40 0x7fa7240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.978+0000 7fa73d0e0640 1 -- 172.21.15.6:0/4007924926 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa724003520 con 0x7fa7240039e0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.979+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7240039e0 0x7fa724003e00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009a90 tx=0x7fa724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a6a17a3aced46c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.979+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c039070 con 0x7fa7240039e0 2024-03-18T14:44:11.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.979+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa72c02c680 con 0x7fa7240039e0 2024-03-18T14:44:11.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.979+0000 7fa72bfff640 1 -- 172.21.15.6:0/4007924926 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c009690 con 0x7fa7240039e0 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.980+0000 7fa73f344640 1 -- 172.21.15.6:0/4007924926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7240039e0 msgr2=0x7fa724003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.980+0000 7fa73f344640 1 --2- 172.21.15.6:0/4007924926 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7240039e0 0x7fa724003e00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009a90 tx=0x7fa724005830 comp rx=0 tx=0).stop 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.980+0000 7fa73d8e1640 1 -- 172.21.15.6:0/4007924926 reap_dead start 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.980+0000 7fa73f344640 1 -- 172.21.15.6:0/4007924926 shutdown_connections 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.980+0000 7fa73f344640 1 -- 172.21.15.6:0/4007924926 >> 172.21.15.6:0/4007924926 conn(0x7fa73806c850 msgr2=0x7fa73806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.980+0000 7fa73f344640 1 -- 172.21.15.6:0/4007924926 shutdown_connections 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.980+0000 7fa73f344640 1 -- 172.21.15.6:0/4007924926 wait complete. 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.981+0000 7fa73f344640 1 Processor -- start 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.981+0000 7fa73f344640 1 -- start start 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73f344640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7240039e0 0x7fa73815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73f344640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73814d0f0 0x7fa7380c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73f344640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7380c97f0 0x7fa7381f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73f344640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa738162800 con 0x7fa7380c97f0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73f344640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa738162520 con 0x7fa7240039e0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73f344640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa7380d1030 con 0x7fa73814d0f0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7380c97f0 0x7fa7381f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7380c97f0 0x7fa7381f4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38026/0 (socket says 172.21.15.6:38026) 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 -- 172.21.15.6:0/1095437371 learned_addr learned my addr 172.21.15.6:0/1095437371 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7240039e0 0x7fa73815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73c8df640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73814d0f0 0x7fa7380c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 -- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73814d0f0 msgr2=0x7fa7380c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73814d0f0 0x7fa7380c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 -- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7240039e0 msgr2=0x7fa73815fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7240039e0 0x7fa73815fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 -- 172.21.15.6:0/1095437371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73815efe0 con 0x7fa7380c97f0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.982+0000 7fa73d8e1640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7380c97f0 0x7fa7381f4000 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa7380cfb10 tx=0x7fa73000fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.983+0000 7fa729ffb640 1 -- 172.21.15.6:0/1095437371 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73001a070 con 0x7fa7380c97f0 2024-03-18T14:44:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.983+0000 7fa729ffb640 1 -- 172.21.15.6:0/1095437371 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa730006d60 con 0x7fa7380c97f0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.983+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa738160e70 con 0x7fa7380c97f0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.983+0000 7fa729ffb640 1 -- 172.21.15.6:0/1095437371 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730003b90 con 0x7fa7380c97f0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.983+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73815f530 con 0x7fa7380c97f0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.984+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa73808afe0 con 0x7fa7380c97f0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.984+0000 7fa729ffb640 1 -- 172.21.15.6:0/1095437371 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa73001d070 con 0x7fa7380c97f0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.985+0000 7fa729ffb640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa71c06bc00 0x7fa71c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.985+0000 7fa729ffb640 1 -- 172.21.15.6:0/1095437371 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fa73008a0b0 con 0x7fa7380c97f0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.985+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa71c06bc00 0x7fa71c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.985+0000 7fa73d0e0640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa71c06bc00 0x7fa71c06e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa72c036000 tx=0x7fa72c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:11.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:11.989+0000 7fa729ffb640 1 -- 172.21.15.6:0/1095437371 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa7300583c0 con 0x7fa7380c97f0 2024-03-18T14:44:12.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.176+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa73804f620 con 0x7fa7380c97f0 2024-03-18T14:44:12.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.177+0000 7fa729ffb640 1 -- 172.21.15.6:0/1095437371 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7fa73005c460 con 0x7fa7380c97f0 2024-03-18T14:44:12.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:12.180 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa71c06bc00 msgr2=0x7fa71c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa71c06bc00 0x7fa71c06e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa72c036000 tx=0x7fa72c030000 comp rx=0 tx=0).stop 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7380c97f0 msgr2=0x7fa7381f4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7380c97f0 0x7fa7381f4000 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa7380cfb10 tx=0x7fa73000fd40 comp rx=0 tx=0).stop 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 shutdown_connections 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa71c06bc00 0x7fa71c06e0c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7380c97f0 0x7fa7381f4000 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa73814d0f0 0x7fa7380c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 --2- 172.21.15.6:0/1095437371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7240039e0 0x7fa73815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 >> 172.21.15.6:0/1095437371 conn(0x7fa73806c850 msgr2=0x7fa7380b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 shutdown_connections 2024-03-18T14:44:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.179+0000 7fa73f344640 1 -- 172.21.15.6:0/1095437371 wait complete. 2024-03-18T14:44:12.193 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T14:44:12.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:44:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.292+0000 7f7126026640 1 Processor -- start 2024-03-18T14:44:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.292+0000 7f7126026640 1 -- start start 2024-03-18T14:44:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.293+0000 7f7126026640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 0x7f7120162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.293+0000 7f7126026640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71200b9660 con 0x7f71200d3210 2024-03-18T14:44:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.293+0000 7f7126026640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71200ce390 con 0x7f71200ced00 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.293+0000 7f7126026640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f71200d2270 con 0x7f712015e020 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.293+0000 7f711ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f71200d3210 0x7f71200d3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45838/0 (socket says 172.21.15.6:45838) 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.293+0000 7f711ffff640 1 -- 172.21.15.6:0/8349966 learned_addr learned my addr 172.21.15.6:0/8349966 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711f7fe640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 0x7f7120162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3815251008 0 0) 0x7f71200b9660 con 0x7f71200d3210 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7108003520 con 0x7f71200d3210 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3467572121 0 0) 0x7f7108003520 con 0x7f71200d3210 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 msgr2=0x7f7120162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 0x7f7120162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 >> v1:172.21.15.92:6789/0 conn(0x7f71200ced00 legacy=0x7f71200d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71200b9900 con 0x7f71200d3210 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f71180019f0 con 0x7f71200d3210 2024-03-18T14:44:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 >> v1:172.21.15.6:6789/0 conn(0x7f71200d3210 legacy=0x7f71200d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71080039e0 0x7f7108003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7108004800 0x7f7108004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 0x7f7108005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f71200d2270 con 0x7f7108004800 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f710800bb80 con 0x7f71080039e0 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f710800bd80 con 0x7f712015e020 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711effd640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7108004800 0x7f7108004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711f7fe640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71080039e0 0x7f7108003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711ffff640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 0x7f7108005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711effd640 1 -- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 msgr2=0x7f7108005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711effd640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f712015e020 0x7f7108005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711effd640 1 -- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71080039e0 msgr2=0x7f7108003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711effd640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71080039e0 0x7f7108003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.294+0000 7f711effd640 1 -- 172.21.15.6:0/8349966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7108003520 con 0x7f7108004800 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.295+0000 7f711effd640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7108004800 0x7f7108004c80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7114001750 tx=0x7f7108005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815daa995361eb77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.295+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114001370 con 0x7f7108004800 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.295+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71140052d0 con 0x7f7108004800 2024-03-18T14:44:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.295+0000 7f711e7fc640 1 -- 172.21.15.6:0/8349966 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114003830 con 0x7f7108004800 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.296+0000 7f7126026640 1 -- 172.21.15.6:0/8349966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7108004800 msgr2=0x7f7108004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.296+0000 7f7126026640 1 --2- 172.21.15.6:0/8349966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7108004800 0x7f7108004c80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7114001750 tx=0x7f7108005830 comp rx=0 tx=0).stop 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.296+0000 7f711ffff640 1 -- 172.21.15.6:0/8349966 reap_dead start 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.296+0000 7f7126026640 1 -- 172.21.15.6:0/8349966 shutdown_connections 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.296+0000 7f7126026640 1 -- 172.21.15.6:0/8349966 >> 172.21.15.6:0/8349966 conn(0x7f7120072ce0 msgr2=0x7f71200730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.296+0000 7f7126026640 1 -- 172.21.15.6:0/8349966 shutdown_connections 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.296+0000 7f7126026640 1 -- 172.21.15.6:0/8349966 wait complete. 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.298+0000 7f7126026640 1 Processor -- start 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.298+0000 7f7126026640 1 -- start start 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f7126026640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7108004800 0x7f7120155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f7126026640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71201f5650 0x7f71200b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f7126026640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71200b7680 0x7f71201f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f7126026640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f71200cf2c0 con 0x7f71201f5650 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f7126026640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f71200ce890 con 0x7f7108004800 2024-03-18T14:44:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f7126026640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f71200cf440 con 0x7f71200b7680 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f711effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71201f5650 0x7f71200b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f711effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71201f5650 0x7f71200b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38050/0 (socket says 172.21.15.6:38050) 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f711effd640 1 -- 172.21.15.6:0/2611588452 learned_addr learned my addr 172.21.15.6:0/2611588452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f711f7fe640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7108004800 0x7f7120155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.299+0000 7f711ffff640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71200b7680 0x7f71201f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f711effd640 1 -- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71200b7680 msgr2=0x7f71201f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f711effd640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71200b7680 0x7f71201f81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f711effd640 1 -- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7108004800 msgr2=0x7f7120155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f711effd640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7108004800 0x7f7120155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f711effd640 1 -- 172.21.15.6:0/2611588452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71200b4980 con 0x7f71201f5650 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f711effd640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71201f5650 0x7f71200b7140 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7114008fd0 tx=0x7f7114001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f7107fff640 1 -- 172.21.15.6:0/2611588452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114041070 con 0x7f71201f5650 2024-03-18T14:44:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f7107fff640 1 -- 172.21.15.6:0/2611588452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7114005480 con 0x7f71201f5650 2024-03-18T14:44:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71200b4c60 con 0x7f71201f5650 2024-03-18T14:44:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f7107fff640 1 -- 172.21.15.6:0/2611588452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114001370 con 0x7f71201f5650 2024-03-18T14:44:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.300+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71200b5220 con 0x7f71201f5650 2024-03-18T14:44:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.301+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f712008afe0 con 0x7f71201f5650 2024-03-18T14:44:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.302+0000 7f7107fff640 1 -- 172.21.15.6:0/2611588452 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7114044070 con 0x7f71201f5650 2024-03-18T14:44:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.302+0000 7f7107fff640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f710c06bc00 0x7f710c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.302+0000 7f7107fff640 1 -- 172.21.15.6:0/2611588452 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f71140ac4e0 con 0x7f71201f5650 2024-03-18T14:44:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.302+0000 7f711f7fe640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f710c06bc00 0x7f710c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.303+0000 7f711f7fe640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f710c06bc00 0x7f710c06e0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f712015efe0 tx=0x7f7110009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.306+0000 7f7107fff640 1 -- 172.21.15.6:0/2611588452 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f711407a7f0 con 0x7f71201f5650 2024-03-18T14:44:12.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.528+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71200b7a60 con 0x7f71201f5650 2024-03-18T14:44:12.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.528+0000 7f7107fff640 1 -- 172.21.15.6:0/2611588452 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7f711407e890 con 0x7f71201f5650 2024-03-18T14:44:12.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:12.531 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/1499402578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1499402578},{"type":"v1","addr":"172.21.15.92:6836","nonce":1499402578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6022,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/300544617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":300544617},{"type":"v1","addr":"172.21.15.6:6835","nonce":300544617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/175151750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":175151750},{"type":"v1","addr":"172.21.15.92:6837","nonce":175151750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:43:41.338763+0000","modified":"2024-03-18T14:43:44.344988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6225},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6225":{"gid":6225,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/3070832840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3070832840},{"type":"v1","addr":"172.21.15.6:6834","nonce":3070832840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6225,"qdb_cluster":[6225]},"id":3}]} 2024-03-18T14:44:12.531 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f710c06bc00 msgr2=0x7f710c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f710c06bc00 0x7f710c06e0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f712015efe0 tx=0x7f7110009190 comp rx=0 tx=0).stop 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71201f5650 msgr2=0x7f71200b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71201f5650 0x7f71200b7140 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7114008fd0 tx=0x7f7114001120 comp rx=0 tx=0).stop 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 shutdown_connections 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f710c06bc00 0x7f710c06e0c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f71200b7680 0x7f71201f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71201f5650 0x7f71200b7140 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 --2- 172.21.15.6:0/2611588452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7108004800 0x7f7120155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 >> 172.21.15.6:0/2611588452 conn(0x7f7120072ce0 msgr2=0x7f712015c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:12.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 shutdown_connections 2024-03-18T14:44:12.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.531+0000 7f7126026640 1 -- 172.21.15.6:0/2611588452 wait complete. 2024-03-18T14:44:12.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:44:12.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.644+0000 7fadc4a12640 1 Processor -- start 2024-03-18T14:44:12.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.644+0000 7fadc4a12640 1 -- start start 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadc4a12640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fadc015ab10 0x7fadc00b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadc4a12640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fadc00ae840 0x7fadc00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadc4a12640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadc006f120 con 0x7fadc00cb680 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadc4a12640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fadc006fca0 con 0x7fadc00ae840 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadc4a12640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fadc006f400 con 0x7fadc015ab10 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbdd9b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fadc00ae840 0x7fadc00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fadc00cb680 0x7fadc00cbac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45846/0 (socket says 172.21.15.6:45846) 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbed9d640 1 -- 172.21.15.6:0/1069985220 learned_addr learned my addr 172.21.15.6:0/1069985220 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbe59c640 1 --2- 172.21.15.6:0/1069985220 >> v2:172.21.15.92:3300/0 conn(0x7fadc015ab10 0x7fadc00b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbdd9b640 1 -- 172.21.15.6:0/1069985220 >> v2:172.21.15.92:3300/0 conn(0x7fadc015ab10 msgr2=0x7fadc00b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbd59a640 1 -- 172.21.15.6:0/1069985220 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729004880 0 0) 0x7fadc006f120 con 0x7fadc00cb680 2024-03-18T14:44:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbdd9b640 1 --2- 172.21.15.6:0/1069985220 >> v2:172.21.15.92:3300/0 conn(0x7fadc015ab10 0x7fadc00b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.645+0000 7fadbdd9b640 1 -- 172.21.15.6:0/1069985220 >> v1:172.21.15.6:6789/0 conn(0x7fadc00cb680 legacy=0x7fadc00cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.646+0000 7fadbdd9b640 1 -- 172.21.15.6:0/1069985220 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc006f6e0 con 0x7fadc00ae840 2024-03-18T14:44:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.646+0000 7fadbe59c640 1 --2- 172.21.15.6:0/1069985220 >> v2:172.21.15.92:3300/0 conn(0x7fadc015ab10 0x7fadc00b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.646+0000 7fadbdd9b640 1 --2- 172.21.15.6:0/1069985220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc00ae840 0x7fadc00cd450 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fadc00ccee0 tx=0x7fada802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316bd1d6758c217f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.646+0000 7fadbd59a640 1 -- 172.21.15.6:0/1069985220 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8035020 con 0x7fadc00ae840 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.646+0000 7fadbd59a640 1 -- 172.21.15.6:0/1069985220 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada8031830 con 0x7fadc00ae840 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.646+0000 7fadbd59a640 1 -- 172.21.15.6:0/1069985220 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8031c00 con 0x7fadc00ae840 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 -- 172.21.15.6:0/1069985220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc00ae840 msgr2=0x7fadc00cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 --2- 172.21.15.6:0/1069985220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc00ae840 0x7fadc00cd450 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fadc00ccee0 tx=0x7fada802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 -- 172.21.15.6:0/1069985220 shutdown_connections 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 --2- 172.21.15.6:0/1069985220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc00ae840 0x7fadc00cd450 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 --2- 172.21.15.6:0/1069985220 >> v2:172.21.15.92:3300/0 conn(0x7fadc015ab10 0x7fadc00b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 -- 172.21.15.6:0/1069985220 >> 172.21.15.6:0/1069985220 conn(0x7fadc00b1470 msgr2=0x7fadc00b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 -- 172.21.15.6:0/1069985220 shutdown_connections 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.647+0000 7fadc4a12640 1 -- 172.21.15.6:0/1069985220 wait complete. 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.648+0000 7fadc4a12640 1 Processor -- start 2024-03-18T14:44:12.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.648+0000 7fadc4a12640 1 -- start start 2024-03-18T14:44:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadc4a12640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadc00ae840 0x7fadc014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadc4a12640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadc00cb680 0x7fadc014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadc4a12640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc015ab10 0x7fadc015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadc4a12640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fadc015aef0 con 0x7fadc015ab10 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadc4a12640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fadc00b56f0 con 0x7fadc00ae840 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadc4a12640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fadc00cfce0 con 0x7fadc00cb680 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadbed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc015ab10 0x7fadc015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadbed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc015ab10 0x7fadc015b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38076/0 (socket says 172.21.15.6:38076) 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadbed9d640 1 -- 172.21.15.6:0/2986747893 learned_addr learned my addr 172.21.15.6:0/2986747893 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadbe59c640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadc00ae840 0x7fadc014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadbdd9b640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadc00cb680 0x7fadc014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadbed9d640 1 -- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadc00cb680 msgr2=0x7fadc014ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.649+0000 7fadbed9d640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadc00cb680 0x7fadc014ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadbed9d640 1 -- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadc00ae840 msgr2=0x7fadc014e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadbed9d640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadc00ae840 0x7fadc014e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadbed9d640 1 -- 172.21.15.6:0/2986747893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc014e790 con 0x7fadc015ab10 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadbe59c640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadc00ae840 0x7fadc014e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadbed9d640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc015ab10 0x7fadc015b200 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fadb800c7e0 tx=0x7fadb800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadb6ffd640 1 -- 172.21.15.6:0/2986747893 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8018070 con 0x7fadc015ab10 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadb6ffd640 1 -- 172.21.15.6:0/2986747893 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb800a640 con 0x7fadc015ab10 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc014bcb0 con 0x7fadc015ab10 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadb6ffd640 1 -- 172.21.15.6:0/2986747893 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb800aa30 con 0x7fadc015ab10 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.650+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc014c550 con 0x7fadc015ab10 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.651+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc0073310 con 0x7fadc015ab10 2024-03-18T14:44:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.652+0000 7fadb6ffd640 1 -- 172.21.15.6:0/2986747893 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fadb800abd0 con 0x7fadc015ab10 2024-03-18T14:44:12.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.652+0000 7fadb6ffd640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fada006bbd0 0x7fada006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.652+0000 7fadb6ffd640 1 -- 172.21.15.6:0/2986747893 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fadb808a380 con 0x7fadc015ab10 2024-03-18T14:44:12.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.652+0000 7fadbe59c640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fada006bbd0 0x7fada006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.653+0000 7fadbe59c640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fada006bbd0 0x7fada006e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fadb000a990 tx=0x7fadb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.657+0000 7fadb6ffd640 1 -- 172.21.15.6:0/2986747893 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fadc014e790 con 0x7fadc015ab10 2024-03-18T14:44:12.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.877+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadc004f620 con 0x7fadc015ab10 2024-03-18T14:44:12.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.877+0000 7fadb6ffd640 1 -- 172.21.15.6:0/2986747893 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5169 (secure 0 0 0) 0x7fadb8058c60 con 0x7fadc015ab10 2024-03-18T14:44:12.880 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:12.880 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/1499402578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1499402578},{"type":"v1","addr":"172.21.15.92:6836","nonce":1499402578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6022,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/300544617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":300544617},{"type":"v1","addr":"172.21.15.6:6835","nonce":300544617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/175151750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":175151750},{"type":"v1","addr":"172.21.15.92:6837","nonce":175151750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T14:43:41.338763+0000","modified":"2024-03-18T14:43:44.344988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6225},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6225":{"gid":6225,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/3070832840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3070832840},{"type":"v1","addr":"172.21.15.6:6834","nonce":3070832840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6225,"qdb_cluster":[6225]},"id":3}]} 2024-03-18T14:44:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fada006bbd0 msgr2=0x7fada006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fada006bbd0 0x7fada006e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fadb000a990 tx=0x7fadb0008040 comp rx=0 tx=0).stop 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc015ab10 msgr2=0x7fadc015b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc015ab10 0x7fadc015b200 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fadb800c7e0 tx=0x7fadb800cca0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 shutdown_connections 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fada006bbd0 0x7fada006e090 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc015ab10 0x7fadc015b200 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadc00cb680 0x7fadc014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 --2- 172.21.15.6:0/2986747893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadc00ae840 0x7fadc014e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 >> 172.21.15.6:0/2986747893 conn(0x7fadc00b1470 msgr2=0x7fadc00b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:12.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 shutdown_connections 2024-03-18T14:44:12.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.879+0000 7fadc4a12640 1 -- 172.21.15.6:0/2986747893 wait complete. 2024-03-18T14:44:12.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:44:12.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.993+0000 7eff45b59640 1 Processor -- start 2024-03-18T14:44:12.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.993+0000 7eff45b59640 1 -- start start 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff45b59640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7eff4015e170 0x7eff400b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff45b59640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff400d0d90 0x7eff400b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff45b59640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff400cef50 con 0x7eff400d3330 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff45b59640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7eff400b9b90 con 0x7eff4015e170 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff45b59640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7eff400cf230 con 0x7eff400d0d90 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7eff4015e170 0x7eff400b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3f7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7eff4015e170 0x7eff400b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38080/0 (socket says 172.21.15.6:38080) 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3f7fe640 1 -- 172.21.15.6:0/3776464240 learned_addr learned my addr 172.21.15.6:0/3776464240 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3ffff640 1 --2- 172.21.15.6:0/3776464240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff400d0d90 0x7eff400b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3f7fe640 1 -- 172.21.15.6:0/3776464240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff400d0d90 msgr2=0x7eff400b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3f7fe640 1 --2- 172.21.15.6:0/3776464240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff400d0d90 0x7eff400b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3f7fe640 1 -- 172.21.15.6:0/3776464240 >> v1:172.21.15.92:6789/0 conn(0x7eff400d3330 legacy=0x7eff400d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3f7fe640 1 -- 172.21.15.6:0/3776464240 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff400d2b30 con 0x7eff4015e170 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.995+0000 7eff3ffff640 1 --2- 172.21.15.6:0/3776464240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff400d0d90 0x7eff400b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.996+0000 7eff3f7fe640 1 --2- 172.21.15.6:0/3776464240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff4015e170 0x7eff400b9650 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7eff28009870 tx=0x7eff2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d921a3dd7a9725bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.996+0000 7eff3e7fc640 1 -- 172.21.15.6:0/3776464240 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28035020 con 0x7eff4015e170 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.996+0000 7eff3e7fc640 1 -- 172.21.15.6:0/3776464240 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff28031640 con 0x7eff4015e170 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.996+0000 7eff3e7fc640 1 -- 172.21.15.6:0/3776464240 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2802d040 con 0x7eff4015e170 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.996+0000 7eff45b59640 1 -- 172.21.15.6:0/3776464240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff4015e170 msgr2=0x7eff400b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.996+0000 7eff45b59640 1 --2- 172.21.15.6:0/3776464240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff4015e170 0x7eff400b9650 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7eff28009870 tx=0x7eff2802b8f0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.997+0000 7eff45b59640 1 -- 172.21.15.6:0/3776464240 shutdown_connections 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.997+0000 7eff45b59640 1 --2- 172.21.15.6:0/3776464240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff400d0d90 0x7eff400b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.997+0000 7eff45b59640 1 --2- 172.21.15.6:0/3776464240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff4015e170 0x7eff400b9650 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.997+0000 7eff45b59640 1 -- 172.21.15.6:0/3776464240 >> 172.21.15.6:0/3776464240 conn(0x7eff4006c900 msgr2=0x7eff4006cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.997+0000 7eff45b59640 1 -- 172.21.15.6:0/3776464240 shutdown_connections 2024-03-18T14:44:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.997+0000 7eff45b59640 1 -- 172.21.15.6:0/3776464240 wait complete. 2024-03-18T14:44:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.998+0000 7eff45b59640 1 Processor -- start 2024-03-18T14:44:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.998+0000 7eff45b59640 1 -- start start 2024-03-18T14:44:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff400d0d90 0x7eff400cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff400d3330 0x7eff4014a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff4015e170 0x7eff401e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7eff400ceb70 con 0x7eff400d0d90 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7eff400d2820 con 0x7eff400d3330 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7eff400cecf0 con 0x7eff4015e170 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff400d0d90 0x7eff400cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff400d0d90 0x7eff400cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38082/0 (socket says 172.21.15.6:38082) 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 -- 172.21.15.6:0/621888419 learned_addr learned my addr 172.21.15.6:0/621888419 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3ffff640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff4015e170 0x7eff401e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3effd640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff400d3330 0x7eff4014a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 -- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff4015e170 msgr2=0x7eff401e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff4015e170 0x7eff401e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 -- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff400d3330 msgr2=0x7eff4014a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff400d3330 0x7eff4014a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 -- 172.21.15.6:0/621888419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4014a5e0 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff3f7fe640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff400d0d90 0x7eff400cb0f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7eff2802b930 tx=0x7eff280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff37fff640 1 -- 172.21.15.6:0/621888419 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28040070 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff37fff640 1 -- 172.21.15.6:0/621888419 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff28039c10 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff37fff640 1 -- 172.21.15.6:0/621888419 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28035040 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4014a960 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:12.999+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff4014af20 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.001+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4008afe0 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.001+0000 7eff37fff640 1 -- 172.21.15.6:0/621888419 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7eff2804b060 con 0x7eff400d0d90 2024-03-18T14:44:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.001+0000 7eff37fff640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7eff2006bbd0 0x7eff2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.002+0000 7eff37fff640 1 -- 172.21.15.6:0/621888419 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7eff280ac770 con 0x7eff400d0d90 2024-03-18T14:44:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.002+0000 7eff3effd640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7eff2006bbd0 0x7eff2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.002+0000 7eff3effd640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7eff2006bbd0 0x7eff2006e090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7eff40161320 tx=0x7eff30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.006+0000 7eff37fff640 1 -- 172.21.15.6:0/621888419 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7eff4014a5e0 con 0x7eff400d0d90 2024-03-18T14:44:13.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.189+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff4004f620 con 0x7eff400d0d90 2024-03-18T14:44:13.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.190+0000 7eff37fff640 1 -- 172.21.15.6:0/621888419 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16842 (secure 0 0 0) 0x7eff2802f300 con 0x7eff400d0d90 2024-03-18T14:44:13.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:13.193 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":56,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:43:42.967094+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-18T14:43:39.597879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-18T14:43:40.299491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7eff2006bbd0 msgr2=0x7eff2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7eff2006bbd0 0x7eff2006e090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7eff40161320 tx=0x7eff30009040 comp rx=0 tx=0).stop 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff400d0d90 msgr2=0x7eff400cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff400d0d90 0x7eff400cb0f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7eff2802b930 tx=0x7eff280046c0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 shutdown_connections 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7eff2006bbd0 0x7eff2006e090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff4015e170 0x7eff401e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff400d3330 0x7eff4014a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 --2- 172.21.15.6:0/621888419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff400d0d90 0x7eff400cb0f0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.192+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 >> 172.21.15.6:0/621888419 conn(0x7eff4006c900 msgr2=0x7eff400d04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:13.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.193+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 shutdown_connections 2024-03-18T14:44:13.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.193+0000 7eff45b59640 1 -- 172.21.15.6:0/621888419 wait complete. 2024-03-18T14:44:13.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T14:44:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.307+0000 7f7405663640 1 Processor -- start 2024-03-18T14:44:13.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.307+0000 7f7405663640 1 -- start start 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f7405663640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f74000d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f7405663640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7400161a10 0x7f74000d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f7405663640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74000d20e0 con 0x7f740015dfb0 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f7405663640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f740015e390 con 0x7f7400161a10 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f7405663640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f7400161f40 con 0x7f74000cf080 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f73ff7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7400161a10 0x7f74000d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f73ff7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7400161a10 0x7f74000d2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38090/0 (socket says 172.21.15.6:38090) 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f73ff7fe640 1 -- 172.21.15.6:0/1268982550 learned_addr learned my addr 172.21.15.6:0/1268982550 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f73fe7fc640 1 --2- 172.21.15.6:0/1268982550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f74000d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f73ff7fe640 1 -- 172.21.15.6:0/1268982550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 msgr2=0x7f74000d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:13.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f73ff7fe640 1 --2- 172.21.15.6:0/1268982550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f74000d25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.308+0000 7f73ff7fe640 1 -- 172.21.15.6:0/1268982550 >> v1:172.21.15.92:6789/0 conn(0x7f740015dfb0 legacy=0x7f74000ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.309+0000 7f73ff7fe640 1 -- 172.21.15.6:0/1268982550 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7400162220 con 0x7f7400161a10 2024-03-18T14:44:13.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f73ff7fe640 1 --2- 172.21.15.6:0/1268982550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7400161a10 0x7f74000d2b00 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009e50 tx=0x7f73f002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d57d4e08d5422cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f73fdffb640 1 -- 172.21.15.6:0/1268982550 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f002cb00 con 0x7f7400161a10 2024-03-18T14:44:13.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f73fdffb640 1 -- 172.21.15.6:0/1268982550 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f002cca0 con 0x7f7400161a10 2024-03-18T14:44:13.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f73fdffb640 1 -- 172.21.15.6:0/1268982550 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0034610 con 0x7f7400161a10 2024-03-18T14:44:13.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 -- 172.21.15.6:0/1268982550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7400161a10 msgr2=0x7f74000d2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 --2- 172.21.15.6:0/1268982550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7400161a10 0x7f74000d2b00 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009e50 tx=0x7f73f002bf60 comp rx=0 tx=0).stop 2024-03-18T14:44:13.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 -- 172.21.15.6:0/1268982550 shutdown_connections 2024-03-18T14:44:13.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 --2- 172.21.15.6:0/1268982550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7400161a10 0x7f74000d2b00 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 --2- 172.21.15.6:0/1268982550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f74000d25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 -- 172.21.15.6:0/1268982550 >> 172.21.15.6:0/1268982550 conn(0x7f740006c850 msgr2=0x7f740006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:13.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 -- 172.21.15.6:0/1268982550 shutdown_connections 2024-03-18T14:44:13.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.310+0000 7f7405663640 1 -- 172.21.15.6:0/1268982550 wait complete. 2024-03-18T14:44:13.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 Processor -- start 2024-03-18T14:44:13.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 -- start start 2024-03-18T14:44:13.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f740014fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f740015dfb0 0x7f740014de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7400161a10 0x7f740015fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f74000d1010 con 0x7f740015dfb0 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74001627e0 con 0x7f7400161a10 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.312+0000 7f7405663640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f74000d1190 con 0x7f74000cf080 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f740015dfb0 0x7f740014de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f740015dfb0 0x7f740014de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38092/0 (socket says 172.21.15.6:38092) 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 -- 172.21.15.6:0/2761253591 learned_addr learned my addr 172.21.15.6:0/2761253591 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73ff7fe640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7400161a10 0x7f740015fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73feffd640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f740014fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 -- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 msgr2=0x7f740014fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f740014fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 -- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7400161a10 msgr2=0x7f740015fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7400161a10 0x7f740015fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 -- 172.21.15.6:0/2761253591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740015ec00 con 0x7f740015dfb0 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73fe7fc640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f740015dfb0 0x7f740014de60 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f73e800d800 tx=0x7f73e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73f77fe640 1 -- 172.21.15.6:0/2761253591 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e800b6c0 con 0x7f740015dfb0 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73f77fe640 1 -- 172.21.15.6:0/2761253591 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e8019070 con 0x7f740015dfb0 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7400150190 con 0x7f740015dfb0 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f73f77fe640 1 -- 172.21.15.6:0/2761253591 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e800bac0 con 0x7f740015dfb0 2024-03-18T14:44:13.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.313+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f740015f180 con 0x7f740015dfb0 2024-03-18T14:44:13.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.315+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f740004f5b0 con 0x7f740015dfb0 2024-03-18T14:44:13.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.315+0000 7f73f77fe640 1 -- 172.21.15.6:0/2761253591 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f73e800bc60 con 0x7f740015dfb0 2024-03-18T14:44:13.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.315+0000 7f73f77fe640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73e006bbd0 0x7f73e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.315+0000 7f73f77fe640 1 -- 172.21.15.6:0/2761253591 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f73e808b060 con 0x7f740015dfb0 2024-03-18T14:44:13.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.315+0000 7f73feffd640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73e006bbd0 0x7f73e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.316+0000 7f73feffd640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73e006bbd0 0x7f73e006e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f73f8000f60 tx=0x7f73f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.319+0000 7f73f77fe640 1 -- 172.21.15.6:0/2761253591 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f73e8059a50 con 0x7f740015dfb0 2024-03-18T14:44:13.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.542+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7400159920 con 0x7f740015dfb0 2024-03-18T14:44:13.545 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-18T14:44:13.543+0000 7f3765211640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T14:44:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.544+0000 7f73f77fe640 1 -- 172.21.15.6:0/2761253591 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7f73e8059350 con 0x7f740015dfb0 2024-03-18T14:44:13.547 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73e006bbd0 msgr2=0x7f73e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73e006bbd0 0x7f73e006e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f73f8000f60 tx=0x7f73f8009040 comp rx=0 tx=0).stop 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f740015dfb0 msgr2=0x7f740014de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f740015dfb0 0x7f740014de60 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f73e800d800 tx=0x7f73e800dcc0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 shutdown_connections 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f73e006bbd0 0x7f73e006e090 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7400161a10 0x7f740015fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f740015dfb0 0x7f740014de60 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 --2- 172.21.15.6:0/2761253591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74000cf080 0x7f740014fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 >> 172.21.15.6:0/2761253591 conn(0x7f740006c850 msgr2=0x7f740006cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.547+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 shutdown_connections 2024-03-18T14:44:13.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.548+0000 7f7405663640 1 -- 172.21.15.6:0/2761253591 wait complete. 2024-03-18T14:44:13.551 INFO:tasks.ceph.mds.c.smithi006.stderr: -2785> 2024-03-18T14:43:47.949+0000 7f9fe9ee7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T14:44:13.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T14:44:13.578 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-18T14:44:13.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.663+0000 7f6c752b2640 1 Processor -- start 2024-03-18T14:44:13.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.663+0000 7f6c752b2640 1 -- start start 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c752b2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6c700cf850 0x7f6c700cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c752b2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6c700cd8f0 0x7f6c700b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c752b2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c700cdd00 con 0x7f6c7015ac60 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c752b2640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6c700cf3c0 con 0x7f6c700cf850 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c752b2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f6c700ce590 con 0x7f6c700cd8f0 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c6e7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6c700cf850 0x7f6c700cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c6e7fc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6c700cf850 0x7f6c700cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41328/0 (socket says 172.21.15.6:41328) 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.664+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/3821269895 learned_addr learned my addr 172.21.15.6:0/3821269895 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.665+0000 7f6c6f7fe640 1 --2- 172.21.15.6:0/3821269895 >> v2:172.21.15.92:3300/0 conn(0x7f6c700cd8f0 0x7f6c700b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.665+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/3821269895 >> v2:172.21.15.92:3300/0 conn(0x7f6c700cd8f0 msgr2=0x7f6c700b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:13.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.665+0000 7f6c6e7fc640 1 --2- 172.21.15.6:0/3821269895 >> v2:172.21.15.92:3300/0 conn(0x7f6c700cd8f0 0x7f6c700b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.665+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/3821269895 >> v1:172.21.15.92:6789/0 conn(0x7f6c7015ac60 legacy=0x7f6c700cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:13.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.665+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/3821269895 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c700ce830 con 0x7f6c700cf850 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.665+0000 7f6c6e7fc640 1 --2- 172.21.15.6:0/3821269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700cd3b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6c700d0970 tx=0x7f6c5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e184f04e39e65b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c6dffb640 1 -- 172.21.15.6:0/3821269895 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58035020 con 0x7f6c700cf850 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c6dffb640 1 -- 172.21.15.6:0/3821269895 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c58031870 con 0x7f6c700cf850 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c6dffb640 1 -- 172.21.15.6:0/3821269895 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58031c40 con 0x7f6c700cf850 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 -- 172.21.15.6:0/3821269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 msgr2=0x7f6c700cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 --2- 172.21.15.6:0/3821269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700cd3b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6c700d0970 tx=0x7f6c5802b910 comp rx=0 tx=0).stop 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 -- 172.21.15.6:0/3821269895 shutdown_connections 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 --2- 172.21.15.6:0/3821269895 >> v2:172.21.15.92:3300/0 conn(0x7f6c700cd8f0 0x7f6c700b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 --2- 172.21.15.6:0/3821269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700cd3b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 -- 172.21.15.6:0/3821269895 >> 172.21.15.6:0/3821269895 conn(0x7f6c70099a50 msgr2=0x7f6c700d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 -- 172.21.15.6:0/3821269895 shutdown_connections 2024-03-18T14:44:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.666+0000 7f6c752b2640 1 -- 172.21.15.6:0/3821269895 wait complete. 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.667+0000 7f6c752b2640 1 Processor -- start 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.667+0000 7f6c752b2640 1 -- start start 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c752b2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c700cd8f0 0x7f6c70153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c752b2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c752b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6c7015ac60 0x7f6c700ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c752b2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6c700cf050 con 0x7f6c700cf850 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c752b2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6c700cead0 con 0x7f6c700cd8f0 2024-03-18T14:44:13.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c752b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f6c7015b040 con 0x7f6c7015ac60 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41338/0 (socket says 172.21.15.6:41338) 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/1647416825 learned_addr learned my addr 172.21.15.6:0/1647416825 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6effd640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c700cd8f0 0x7f6c70153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6f7fe640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6c7015ac60 0x7f6c700ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6c7015ac60 msgr2=0x7f6c700ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6c7015ac60 0x7f6c700ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c700cd8f0 msgr2=0x7f6c70153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c700cd8f0 0x7f6c70153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.668+0000 7f6c6e7fc640 1 -- 172.21.15.6:0/1647416825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c700b60d0 con 0x7f6c700cf850 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.669+0000 7f6c6effd640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c700cd8f0 0x7f6c70153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.669+0000 7f6c6e7fc640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700b4660 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6c58000c00 tx=0x7f6c580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.669+0000 7f6c677fe640 1 -- 172.21.15.6:0/1647416825 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58040070 con 0x7f6c700cf850 2024-03-18T14:44:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.669+0000 7f6c677fe640 1 -- 172.21.15.6:0/1647416825 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c58030d00 con 0x7f6c700cf850 2024-03-18T14:44:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.669+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c700b3620 con 0x7f6c700cf850 2024-03-18T14:44:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.669+0000 7f6c677fe640 1 -- 172.21.15.6:0/1647416825 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58035040 con 0x7f6c700cf850 2024-03-18T14:44:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.669+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c700b3e60 con 0x7f6c700cf850 2024-03-18T14:44:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.670+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c700751b0 con 0x7f6c700cf850 2024-03-18T14:44:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.670+0000 7f6c677fe640 1 -- 172.21.15.6:0/1647416825 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6c5804b060 con 0x7f6c700cf850 2024-03-18T14:44:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.671+0000 7f6c677fe640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6c5006bbd0 0x7f6c5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.671+0000 7f6c677fe640 1 -- 172.21.15.6:0/1647416825 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6187+0+0 (secure 0 0 0) 0x7f6c580aca40 con 0x7f6c700cf850 2024-03-18T14:44:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.671+0000 7f6c6effd640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6c5006bbd0 0x7f6c5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.671+0000 7f6c6effd640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6c5006bbd0 0x7f6c5006e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6c68006fd0 tx=0x7f6c68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:13.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.675+0000 7f6c677fe640 1 -- 172.21.15.6:0/1647416825 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6c700b3620 con 0x7f6c700cf850 2024-03-18T14:44:13.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:13.896+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6c70099fa0 con 0x7f6c700cf850 2024-03-18T14:44:14.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.548+0000 7f6c677fe640 1 -- 172.21.15.6:0/1647416825 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f6c580ac0a0 con 0x7f6c700cf850 2024-03-18T14:44:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6c5006bbd0 msgr2=0x7f6c5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6c5006bbd0 0x7f6c5006e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6c68006fd0 tx=0x7f6c68008040 comp rx=0 tx=0).stop 2024-03-18T14:44:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 msgr2=0x7f6c700b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700b4660 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6c58000c00 tx=0x7f6c580046c0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 shutdown_connections 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f6c5006bbd0 0x7f6c5006e090 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6c7015ac60 0x7f6c700ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c700cf850 0x7f6c700b4660 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 --2- 172.21.15.6:0/1647416825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c700cd8f0 0x7f6c70153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 >> 172.21.15.6:0/1647416825 conn(0x7f6c70099a50 msgr2=0x7f6c7015d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 shutdown_connections 2024-03-18T14:44:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.551+0000 7f6c752b2640 1 -- 172.21.15.6:0/1647416825 wait complete. 2024-03-18T14:44:14.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T14:44:14.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.709+0000 7f10f8ec6640 1 Processor -- start 2024-03-18T14:44:14.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10f8ec6640 1 -- start start 2024-03-18T14:44:14.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10f8ec6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 0x7f10f40b95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10f8ec6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f10f406c730 0x7f10f406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10f8ec6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f40d22c0 con 0x7f10f40d1de0 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10f8ec6640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f10f40d25a0 con 0x7f10f406c730 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10f8ec6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f10f40d2880 con 0x7f10f415e0b0 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10ebfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f10f40d1de0 0x7f10f40d3360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55480/0 (socket says 172.21.15.6:55480) 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.710+0000 7f10ebfff640 1 -- 172.21.15.6:0/3549519007 learned_addr learned my addr 172.21.15.6:0/3549519007 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f259c640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 0x7f10f40b95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f2d9d640 1 --2- 172.21.15.6:0/3549519007 >> v2:172.21.15.92:3300/0 conn(0x7f10f406c730 0x7f10f406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927249292 0 0) 0x7f10f40d22c0 con 0x7f10f40d1de0 2024-03-18T14:44:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e0003520 con 0x7f10f40d1de0 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266973232 0 0) 0x7f10e0003520 con 0x7f10f40d1de0 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 msgr2=0x7f10f40b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 0x7f10f40b95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 >> v2:172.21.15.92:3300/0 conn(0x7f10f406c730 msgr2=0x7f10f406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 --2- 172.21.15.6:0/3549519007 >> v2:172.21.15.92:3300/0 conn(0x7f10f406c730 0x7f10f406cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f40d2b60 con 0x7f10f40d1de0 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f10dc001670 con 0x7f10f40d1de0 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 >> v1:172.21.15.6:6789/0 conn(0x7f10f40d1de0 legacy=0x7f10f40d3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e00039b0 0x7f10e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 0x7f10e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e0004d40 0x7f10e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f10f40d25a0 con 0x7f10e00039b0 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10f40d2880 con 0x7f10e0004d40 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f10e000b9c0 con 0x7f10f415e0b0 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f259c640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e00039b0 0x7f10e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f2d9d640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e0004d40 0x7f10e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10ebfff640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 0x7f10e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f259c640 1 -- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 msgr2=0x7f10e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f259c640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f415e0b0 0x7f10e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f259c640 1 -- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e0004d40 msgr2=0x7f10e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f259c640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e0004d40 0x7f10e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.711+0000 7f10f259c640 1 -- 172.21.15.6:0/3549519007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e0003520 con 0x7f10e00039b0 2024-03-18T14:44:14.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.712+0000 7f10f259c640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e00039b0 0x7f10e0003dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009ac0 tx=0x7f10e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2442beeea35965 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.712+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4039070 con 0x7f10e00039b0 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.712+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e402c6b0 con 0x7f10e00039b0 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.712+0000 7f10f1d9b640 1 -- 172.21.15.6:0/3549519007 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4009690 con 0x7f10e00039b0 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.713+0000 7f10f8ec6640 1 -- 172.21.15.6:0/3549519007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e00039b0 msgr2=0x7f10e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.713+0000 7f10f8ec6640 1 --2- 172.21.15.6:0/3549519007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e00039b0 0x7f10e0003dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009ac0 tx=0x7f10e0005830 comp rx=0 tx=0).stop 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.713+0000 7f10f2d9d640 1 -- 172.21.15.6:0/3549519007 reap_dead start 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.713+0000 7f10f8ec6640 1 -- 172.21.15.6:0/3549519007 shutdown_connections 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.713+0000 7f10f8ec6640 1 -- 172.21.15.6:0/3549519007 >> 172.21.15.6:0/3549519007 conn(0x7f10f4072ce0 msgr2=0x7f10f40730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.713+0000 7f10f8ec6640 1 -- 172.21.15.6:0/3549519007 shutdown_connections 2024-03-18T14:44:14.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.713+0000 7f10f8ec6640 1 -- 172.21.15.6:0/3549519007 wait complete. 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.714+0000 7f10f8ec6640 1 Processor -- start 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.714+0000 7f10f8ec6640 1 -- start start 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10f8ec6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f40b7f50 0x7f10f414fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10f8ec6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10f40b8330 0x7f10f4150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10f8ec6640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f40cad50 0x7f10f40cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10f8ec6640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f10f40b9770 con 0x7f10f40b8330 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10f8ec6640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10f40b95f0 con 0x7f10f40b7f50 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10f8ec6640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f10f40b98f0 con 0x7f10f40cad50 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10f40b8330 0x7f10f4150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10f40b8330 0x7f10f4150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41350/0 (socket says 172.21.15.6:41350) 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 -- 172.21.15.6:0/91557580 learned_addr learned my addr 172.21.15.6:0/91557580 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 -- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f40cad50 msgr2=0x7f10f40cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:14.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f40cad50 0x7f10f40cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 -- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f40b7f50 msgr2=0x7f10f414fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f40b7f50 0x7f10f414fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 -- 172.21.15.6:0/91557580 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f40cbac0 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.715+0000 7f10ebfff640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10f40b8330 0x7f10f4150210 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f10dc007310 tx=0x7f10dc0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.716+0000 7f10eaffd640 1 -- 172.21.15.6:0/91557580 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc00fa00 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.716+0000 7f10eaffd640 1 -- 172.21.15.6:0/91557580 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10dc018070 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.716+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f40cbda0 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.716+0000 7f10eaffd640 1 -- 172.21.15.6:0/91557580 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc00fe20 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.716+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f40cc360 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.717+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f408afe0 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.717+0000 7f10eaffd640 1 -- 172.21.15.6:0/91557580 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f10dc01d460 con 0x7f10f40b8330 2024-03-18T14:44:14.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.717+0000 7f10eaffd640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10d406bc00 0x7f10d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:14.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.717+0000 7f10eaffd640 1 -- 172.21.15.6:0/91557580 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6187+0+0 (secure 0 0 0) 0x7f10dc08ad00 con 0x7f10f40b8330 2024-03-18T14:44:14.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.718+0000 7f10f259c640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10d406bc00 0x7f10d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:14.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.718+0000 7f10f259c640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10d406bc00 0x7f10d406e0c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f10e4036000 tx=0x7f10e402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:14.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.722+0000 7f10eaffd640 1 -- 172.21.15.6:0/91557580 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f10dc0596a0 con 0x7f10f40b8330 2024-03-18T14:44:14.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:14.904+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f10f404f620 con 0x7f10f40b8330 2024-03-18T14:44:15.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.552+0000 7f10eaffd640 1 -- 172.21.15.6:0/91557580 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) ==== 181+0+0 (secure 0 0 0) 0x7f10dc058fa0 con 0x7f10f40b8330 2024-03-18T14:44:15.554 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10d406bc00 msgr2=0x7f10d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10d406bc00 0x7f10d406e0c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f10e4036000 tx=0x7f10e402e000 comp rx=0 tx=0).stop 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10f40b8330 msgr2=0x7f10f4150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10f40b8330 0x7f10f4150210 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f10dc007310 tx=0x7f10dc0077d0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 shutdown_connections 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10d406bc00 0x7f10d406e0c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10f40cad50 0x7f10f40cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10f40b8330 0x7f10f4150210 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 --2- 172.21.15.6:0/91557580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f40b7f50 0x7f10f414fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 >> 172.21.15.6:0/91557580 conn(0x7f10f4072ce0 msgr2=0x7f10f40c8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 shutdown_connections 2024-03-18T14:44:15.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.555+0000 7f10f8ec6640 1 -- 172.21.15.6:0/91557580 wait complete. 2024-03-18T14:44:15.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T14:44:15.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.682+0000 7f16df168640 1 Processor -- start 2024-03-18T14:44:15.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.682+0000 7f16df168640 1 -- start start 2024-03-18T14:44:15.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.682+0000 7f16df168640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16d815e000 0x7f16d80b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.682+0000 7f16df168640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.682+0000 7f16df168640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d80b1ca0 con 0x7f16d80ce3b0 2024-03-18T14:44:15.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.682+0000 7f16df168640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f16d80b28b0 con 0x7f16d815e000 2024-03-18T14:44:15.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.682+0000 7f16df168640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f16d80d23d0 con 0x7f16d80d1a50 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16dcf04640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16d815e000 0x7f16d80b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16d7fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16dcf04640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16d815e000 0x7f16d80b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50014/0 (socket says 172.21.15.6:50014) 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16dcf04640 1 -- 172.21.15.6:0/4095696419 learned_addr learned my addr 172.21.15.6:0/4095696419 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16d7fff640 1 --2- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d80d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34576/0 (socket says 172.21.15.6:34576) 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16dcf04640 1 -- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 msgr2=0x7f16d80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16dcf04640 1 --2- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16dcf04640 1 -- 172.21.15.6:0/4095696419 >> v1:172.21.15.92:6789/0 conn(0x7f16d80ce3b0 legacy=0x7f16d80ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16dcf04640 1 -- 172.21.15.6:0/4095696419 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80b1f80 con 0x7f16d815e000 2024-03-18T14:44:15.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.683+0000 7f16d7fff640 1 --2- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:44:15.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.684+0000 7f16dcf04640 1 --2- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d815e000 0x7f16d80b2e00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f16d0009e50 tx=0x7f16d002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e687566be61346 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:15.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.684+0000 7f16d77fe640 1 -- 172.21.15.6:0/4095696419 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d002cb90 con 0x7f16d815e000 2024-03-18T14:44:15.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.684+0000 7f16d77fe640 1 -- 172.21.15.6:0/4095696419 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d002cd30 con 0x7f16d815e000 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.684+0000 7f16d77fe640 1 -- 172.21.15.6:0/4095696419 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0034610 con 0x7f16d815e000 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 -- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d815e000 msgr2=0x7f16d80b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 --2- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d815e000 0x7f16d80b2e00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f16d0009e50 tx=0x7f16d002bff0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 -- 172.21.15.6:0/4095696419 shutdown_connections 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 --2- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 --2- 172.21.15.6:0/4095696419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d815e000 0x7f16d80b2e00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 -- 172.21.15.6:0/4095696419 >> 172.21.15.6:0/4095696419 conn(0x7f16d809ae00 msgr2=0x7f16d808b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 -- 172.21.15.6:0/4095696419 shutdown_connections 2024-03-18T14:44:15.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.685+0000 7f16df168640 1 -- 172.21.15.6:0/4095696419 wait complete. 2024-03-18T14:44:15.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.686+0000 7f16df168640 1 Processor -- start 2024-03-18T14:44:15.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.686+0000 7f16df168640 1 -- start start 2024-03-18T14:44:15.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16df168640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d80ce3b0 0x7f16d815e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16df168640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d8150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16df168640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16d815e000 0x7f16d80cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16df168640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f16d80b2540 con 0x7f16d815e000 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16df168640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f16d815e3e0 con 0x7f16d80ce3b0 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16df168640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f16d80b1210 con 0x7f16d80d1a50 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16dd705640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16d815e000 0x7f16d80cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16dd705640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16d815e000 0x7f16d80cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41364/0 (socket says 172.21.15.6:41364) 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16dd705640 1 -- 172.21.15.6:0/46742807 learned_addr learned my addr 172.21.15.6:0/46742807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16dcf04640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d80ce3b0 0x7f16d815e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16d7fff640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d8150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16dd705640 1 -- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 msgr2=0x7f16d8150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16dd705640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d8150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.687+0000 7f16dd705640 1 -- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d80ce3b0 msgr2=0x7f16d815e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:15.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16dd705640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d80ce3b0 0x7f16d815e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:15.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16dd705640 1 -- 172.21.15.6:0/46742807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80b57d0 con 0x7f16d815e000 2024-03-18T14:44:15.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16dd705640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16d815e000 0x7f16d80cb2f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f16c8001eb0 tx=0x7f16c800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:15.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16d57fa640 1 -- 172.21.15.6:0/46742807 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8017070 con 0x7f16d815e000 2024-03-18T14:44:15.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16d57fa640 1 -- 172.21.15.6:0/46742807 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c80052d0 con 0x7f16d815e000 2024-03-18T14:44:15.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d80b5ab0 con 0x7f16d815e000 2024-03-18T14:44:15.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16d57fa640 1 -- 172.21.15.6:0/46742807 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8003810 con 0x7f16d815e000 2024-03-18T14:44:15.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.688+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d80b6070 con 0x7f16d815e000 2024-03-18T14:44:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.689+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d809fa70 con 0x7f16d815e000 2024-03-18T14:44:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.690+0000 7f16d57fa640 1 -- 172.21.15.6:0/46742807 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f16c801a070 con 0x7f16d815e000 2024-03-18T14:44:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.690+0000 7f16d57fa640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f16c406bbd0 0x7f16c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.690+0000 7f16d57fa640 1 -- 172.21.15.6:0/46742807 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5749+0+0 (secure 0 0 0) 0x7f16c8089e00 con 0x7f16d815e000 2024-03-18T14:44:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.690+0000 7f16dcf04640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f16c406bbd0 0x7f16c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:15.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.691+0000 7f16dcf04640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f16c406bbd0 0x7f16c406e090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f16d0035000 tx=0x7f16d0030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:15.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.694+0000 7f16d57fa640 1 -- 172.21.15.6:0/46742807 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f16c80581e0 con 0x7f16d815e000 2024-03-18T14:44:15.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:15.883+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f16d804f620 con 0x7f16d815e000 2024-03-18T14:44:16.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.554+0000 7f16d57fa640 1 -- 172.21.15.6:0/46742807 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) ==== 169+0+0 (secure 0 0 0) 0x7f16c805c280 con 0x7f16d815e000 2024-03-18T14:44:16.555 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-18T14:44:16.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f16c406bbd0 msgr2=0x7f16c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:16.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f16c406bbd0 0x7f16c406e090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f16d0035000 tx=0x7f16d0030000 comp rx=0 tx=0).stop 2024-03-18T14:44:16.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16d815e000 msgr2=0x7f16d80cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:16.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16d815e000 0x7f16d80cb2f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f16c8001eb0 tx=0x7f16c800c890 comp rx=0 tx=0).stop 2024-03-18T14:44:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 shutdown_connections 2024-03-18T14:44:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f16c406bbd0 0x7f16c406e090 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16d815e000 0x7f16d80cb2f0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d80d1a50 0x7f16d8150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 --2- 172.21.15.6:0/46742807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d80ce3b0 0x7f16d815e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.558+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 >> 172.21.15.6:0/46742807 conn(0x7f16d809ae00 msgr2=0x7f16d80ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.559+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 shutdown_connections 2024-03-18T14:44:16.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.559+0000 7f16df168640 1 -- 172.21.15.6:0/46742807 wait complete. 2024-03-18T14:44:16.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T14:44:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.683+0000 7f13e3836640 1 Processor -- start 2024-03-18T14:44:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.683+0000 7f13e3836640 1 -- start start 2024-03-18T14:44:16.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e3836640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13dc15dfd0 0x7f13dc0b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:16.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e3836640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13dc0b7e00 0x7f13dc0b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e3836640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13dc15e3b0 con 0x7f13dc0b5740 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e3836640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f13dc0b8240 con 0x7f13dc0b7e00 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e3836640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f13dc0b8470 con 0x7f13dc15dfd0 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e1dd3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13dc0b7e00 0x7f13dc0b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e1dd3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13dc0b7e00 0x7f13dc0b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41376/0 (socket says 172.21.15.6:41376) 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e1dd3640 1 -- 172.21.15.6:0/4042767169 learned_addr learned my addr 172.21.15.6:0/4042767169 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e15d2640 1 --2- 172.21.15.6:0/4042767169 >> v2:172.21.15.92:3300/0 conn(0x7f13dc15dfd0 0x7f13dc0b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e1dd3640 1 -- 172.21.15.6:0/4042767169 >> v2:172.21.15.92:3300/0 conn(0x7f13dc15dfd0 msgr2=0x7f13dc0b9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e1dd3640 1 --2- 172.21.15.6:0/4042767169 >> v2:172.21.15.92:3300/0 conn(0x7f13dc15dfd0 0x7f13dc0b9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e1dd3640 1 -- 172.21.15.6:0/4042767169 >> v1:172.21.15.92:6789/0 conn(0x7f13dc0b5740 legacy=0x7f13dc0d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e1dd3640 1 -- 172.21.15.6:0/4042767169 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc0b8750 con 0x7f13dc0b7e00 2024-03-18T14:44:16.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.684+0000 7f13e15d2640 1 --2- 172.21.15.6:0/4042767169 >> v2:172.21.15.92:3300/0 conn(0x7f13dc15dfd0 0x7f13dc0b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:16.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.685+0000 7f13e1dd3640 1 --2- 172.21.15.6:0/4042767169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0b5bd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0d17c0 tx=0x7f13d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8635434e8cd8f566 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:16.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.685+0000 7f13d3fff640 1 -- 172.21.15.6:0/4042767169 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d802ca80 con 0x7f13dc0b7e00 2024-03-18T14:44:16.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.685+0000 7f13d3fff640 1 -- 172.21.15.6:0/4042767169 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d802cc20 con 0x7f13dc0b7e00 2024-03-18T14:44:16.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.685+0000 7f13d3fff640 1 -- 172.21.15.6:0/4042767169 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80343f0 con 0x7f13dc0b7e00 2024-03-18T14:44:16.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 -- 172.21.15.6:0/4042767169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc0b7e00 msgr2=0x7f13dc0b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:16.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 --2- 172.21.15.6:0/4042767169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0b5bd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0d17c0 tx=0x7f13d802bee0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 -- 172.21.15.6:0/4042767169 shutdown_connections 2024-03-18T14:44:16.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 --2- 172.21.15.6:0/4042767169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0b5bd0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 --2- 172.21.15.6:0/4042767169 >> v2:172.21.15.92:3300/0 conn(0x7f13dc15dfd0 0x7f13dc0b9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 -- 172.21.15.6:0/4042767169 >> 172.21.15.6:0/4042767169 conn(0x7f13dc072d10 msgr2=0x7f13dc073120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 -- 172.21.15.6:0/4042767169 shutdown_connections 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.686+0000 7f13e3836640 1 -- 172.21.15.6:0/4042767169 wait complete. 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.687+0000 7f13e3836640 1 Processor -- start 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.687+0000 7f13e3836640 1 -- start start 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e3836640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13dc0b5740 0x7f13dc0c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e3836640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e3836640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc15dfd0 0x7f13dc1520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e3836640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc0b9b90 con 0x7f13dc15dfd0 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e3836640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc0b8d10 con 0x7f13dc0b7e00 2024-03-18T14:44:16.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e3836640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f13dc0d2550 con 0x7f13dc0b5740 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc15dfd0 0x7f13dc1520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc15dfd0 0x7f13dc1520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41390/0 (socket says 172.21.15.6:41390) 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 -- 172.21.15.6:0/2104724689 learned_addr learned my addr 172.21.15.6:0/2104724689 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e15d2640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13dc0b5740 0x7f13dc0c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e0dd1640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 -- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13dc0b5740 msgr2=0x7f13dc0c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13dc0b5740 0x7f13dc0c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 -- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13dc0b7e00 msgr2=0x7f13dc0cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e1dd3640 1 -- 172.21.15.6:0/2104724689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc0c8120 con 0x7f13dc15dfd0 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.688+0000 7f13e0dd1640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.689+0000 7f13e1dd3640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc15dfd0 0x7f13dc1520f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f13d8036040 tx=0x7f13d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.689+0000 7f13d1ffb640 1 -- 172.21.15.6:0/2104724689 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8043020 con 0x7f13dc15dfd0 2024-03-18T14:44:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.689+0000 7f13d1ffb640 1 -- 172.21.15.6:0/2104724689 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d803e450 con 0x7f13dc15dfd0 2024-03-18T14:44:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.689+0000 7f13d1ffb640 1 -- 172.21.15.6:0/2104724689 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d803e860 con 0x7f13dc15dfd0 2024-03-18T14:44:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.689+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13dc0c8400 con 0x7f13dc15dfd0 2024-03-18T14:44:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.689+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13dc0c8ce0 con 0x7f13dc15dfd0 2024-03-18T14:44:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.690+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13dc0751b0 con 0x7f13dc15dfd0 2024-03-18T14:44:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.690+0000 7f13d1ffb640 1 -- 172.21.15.6:0/2104724689 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f13d8046020 con 0x7f13dc15dfd0 2024-03-18T14:44:16.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.691+0000 7f13d1ffb640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f13c006bbd0 0x7f13c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:16.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.691+0000 7f13d1ffb640 1 -- 172.21.15.6:0/2104724689 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f13d80ad2f0 con 0x7f13dc15dfd0 2024-03-18T14:44:16.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.691+0000 7f13e15d2640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f13c006bbd0 0x7f13c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:16.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.691+0000 7f13e15d2640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f13c006bbd0 0x7f13c006e090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f13dc1f6d10 tx=0x7f13d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:16.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.695+0000 7f13d1ffb640 1 -- 172.21.15.6:0/2104724689 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f13d804d030 con 0x7f13dc15dfd0 2024-03-18T14:44:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.878+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f13dc0c8eb0 con 0x7f13dc15dfd0 2024-03-18T14:44:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.881+0000 7f13d1ffb640 1 -- 172.21.15.6:0/2104724689 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f13d807beb0 con 0x7f13dc15dfd0 2024-03-18T14:44:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f13c006bbd0 msgr2=0x7f13c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f13c006bbd0 0x7f13c006e090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f13dc1f6d10 tx=0x7f13d4008040 comp rx=0 tx=0).stop 2024-03-18T14:44:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc15dfd0 msgr2=0x7f13dc1520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc15dfd0 0x7f13dc1520f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f13d8036040 tx=0x7f13d8004a40 comp rx=0 tx=0).stop 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 shutdown_connections 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f13c006bbd0 0x7f13c006e090 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13dc15dfd0 0x7f13dc1520f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13dc0b7e00 0x7f13dc0cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 --2- 172.21.15.6:0/2104724689 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13dc0b5740 0x7f13dc0c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 >> 172.21.15.6:0/2104724689 conn(0x7f13dc072d10 msgr2=0x7f13dc0b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 shutdown_connections 2024-03-18T14:44:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.884+0000 7f13e3836640 1 -- 172.21.15.6:0/2104724689 wait complete. 2024-03-18T14:44:16.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-18T14:44:16.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.996+0000 7fc8c5ad1640 1 Processor -- start 2024-03-18T14:44:16.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.996+0000 7fc8c5ad1640 1 -- start start 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8c5ad1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8c015e240 0x7fc8c006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8c5ad1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c006b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8c5ad1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc8c00d14d0 0x7fc8c00b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8c5ad1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc8c00cb000 con 0x7fc8c015e240 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8c5ad1640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc8c00b1f90 con 0x7fc8c00d14d0 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8c5ad1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c00cce30 con 0x7fc8c00afd00 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8bf7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8c015e240 0x7fc8c006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8bf7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8c015e240 0x7fc8c006b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41402/0 (socket says 172.21.15.6:41402) 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8bf7fe640 1 -- 172.21.15.6:0/3522760825 learned_addr learned my addr 172.21.15.6:0/3522760825 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:17.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8beffd640 1 --2- 172.21.15.6:0/3522760825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c006b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.997+0000 7fc8bffff640 1 --2- 172.21.15.6:0/3522760825 >> v2:172.21.15.92:3300/0 conn(0x7fc8c00d14d0 0x7fc8c00b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.998+0000 7fc8bf7fe640 1 -- 172.21.15.6:0/3522760825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 msgr2=0x7fc8c006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.998+0000 7fc8bf7fe640 1 --2- 172.21.15.6:0/3522760825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c006b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.998+0000 7fc8bf7fe640 1 -- 172.21.15.6:0/3522760825 >> v2:172.21.15.92:3300/0 conn(0x7fc8c00d14d0 msgr2=0x7fc8c00b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.998+0000 7fc8bf7fe640 1 --2- 172.21.15.6:0/3522760825 >> v2:172.21.15.92:3300/0 conn(0x7fc8c00d14d0 0x7fc8c00b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.998+0000 7fc8bf7fe640 1 -- 172.21.15.6:0/3522760825 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c00b1c20 con 0x7fc8c015e240 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.998+0000 7fc8bf7fe640 1 --2- 172.21.15.6:0/3522760825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c006b1c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0072e50 tx=0x7fc8b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc04df1217ce26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.998+0000 7fc8be7fc640 1 -- 172.21.15.6:0/3522760825 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b002e070 con 0x7fc8c015e240 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8be7fc640 1 -- 172.21.15.6:0/3522760825 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b0037070 con 0x7fc8c015e240 2024-03-18T14:44:17.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8be7fc640 1 -- 172.21.15.6:0/3522760825 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0031a00 con 0x7fc8c015e240 2024-03-18T14:44:17.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/3522760825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 msgr2=0x7fc8c006b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:17.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/3522760825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c006b1c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0072e50 tx=0x7fc8b002b880 comp rx=0 tx=0).stop 2024-03-18T14:44:17.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/3522760825 shutdown_connections 2024-03-18T14:44:17.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/3522760825 >> v2:172.21.15.92:3300/0 conn(0x7fc8c00d14d0 0x7fc8c00b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/3522760825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c006b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/3522760825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c006b1c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/3522760825 >> 172.21.15.6:0/3522760825 conn(0x7fc8c00d09b0 msgr2=0x7fc8c00d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/3522760825 shutdown_connections 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:16.999+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/3522760825 wait complete. 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.000+0000 7fc8c5ad1640 1 Processor -- start 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.000+0000 7fc8c5ad1640 1 -- start start 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8c5ad1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c01f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8c5ad1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8c00d14d0 0x7fc8c00c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8c5ad1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c015fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8c5ad1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c00ce700 con 0x7fc8c015e240 2024-03-18T14:44:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8c5ad1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c00ce580 con 0x7fc8c00d14d0 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8c5ad1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c00b06b0 con 0x7fc8c00afd00 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c015fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c015fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41412/0 (socket says 172.21.15.6:41412) 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bffff640 1 -- 172.21.15.6:0/431133966 learned_addr learned my addr 172.21.15.6:0/431133966 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bf7fe640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c01f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8beffd640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8c00d14d0 0x7fc8c00c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bffff640 1 -- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 msgr2=0x7fc8c01f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bffff640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c01f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bffff640 1 -- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8c00d14d0 msgr2=0x7fc8c00c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.001+0000 7fc8bffff640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8c00d14d0 0x7fc8c00c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8bffff640 1 -- 172.21.15.6:0/431133966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c0160d20 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8beffd640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8c00d14d0 0x7fc8c00c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8bffff640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c015fea0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8007e40 tx=0x7fc8b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8a7fff640 1 -- 172.21.15.6:0/431133966 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8014620 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8a7fff640 1 -- 172.21.15.6:0/431133966 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b8019070 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8a7fff640 1 -- 172.21.15.6:0/431133966 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8014a20 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c0161000 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.002+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c00c7180 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.003+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c009e9a0 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.003+0000 7fc8a7fff640 1 -- 172.21.15.6:0/431133966 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc8b8014bc0 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.004+0000 7fc8a7fff640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc8ac06bbd0 0x7fc8ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.004+0000 7fc8a7fff640 1 -- 172.21.15.6:0/431133966 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7fc8b808b240 con 0x7fc8c015e240 2024-03-18T14:44:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.004+0000 7fc8bf7fe640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc8ac06bbd0 0x7fc8ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.005+0000 7fc8bf7fe640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc8ac06bbd0 0x7fc8ac06e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0036000 tx=0x7fc8b003d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.008+0000 7fc8a7fff640 1 -- 172.21.15.6:0/431133966 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc8b8059ea0 con 0x7fc8c015e240 2024-03-18T14:44:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.274+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7fc8c004f620 con 0x7fc8c015e240 2024-03-18T14:44:17.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.561+0000 7fc8a7fff640 1 -- 172.21.15.6:0/431133966 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v177) ==== 134+0+0 (secure 0 0 0) 0x7fc8b80597a0 con 0x7fc8c015e240 2024-03-18T14:44:17.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.563+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc8ac06bbd0 msgr2=0x7fc8ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:17.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.563+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc8ac06bbd0 0x7fc8ac06e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0036000 tx=0x7fc8b003d040 comp rx=0 tx=0).stop 2024-03-18T14:44:17.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.563+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 msgr2=0x7fc8c015fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:17.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.563+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c015fea0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8007e40 tx=0x7fc8b800e4b0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 shutdown_connections 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc8ac06bbd0 0x7fc8ac06e090 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8c015e240 0x7fc8c015fea0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8c00d14d0 0x7fc8c00c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 --2- 172.21.15.6:0/431133966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8c00afd00 0x7fc8c01f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 >> 172.21.15.6:0/431133966 conn(0x7fc8c00d09b0 msgr2=0x7fc8c006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 shutdown_connections 2024-03-18T14:44:17.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:17.564+0000 7fc8c5ad1640 1 -- 172.21.15.6:0/431133966 wait complete. 2024-03-18T14:44:17.578 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2024-03-18T14:44:17.578 INFO:tasks.cephfs_test_runner: 2024-03-18T14:44:17.578 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-18T14:44:17.579 INFO:tasks.cephfs_test_runner: 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner:Ran 2 tests in 114.238s 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner: 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner: 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-18T14:44:17.580 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-18T14:44:17.581 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-18T14:44:17.581 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-18T14:44:17.581 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-18T14:44:17.581 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-18T14:44:17.581 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-18T14:44:17.581 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-18T14:44:17.581 INFO:tasks.cephfs_test_runner: 2024-03-18T14:44:17.581 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-18T14:44:17.807 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=907dc3ee24c34b318112b65a654e49b0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-18T14:44:17.808 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-18T14:44:17.820 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-18T14:44:17.922 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-18T14:44:17.923 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:44:17.923 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:17.942 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2314 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-18T14:44:17.942 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-18T14:44:17.943 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:17.999 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2314 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-18T14:44:17.999 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:44:17.999 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:18.017 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:44:18.017 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-18T14:44:18.017 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T14:44:18.075 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056370,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=381 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-18T14:44:18.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.075 DEBUG:teuthology.orchestra.run.smithi092:> ip netns list 2024-03-18T14:44:18.131 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-18T14:44:18.132 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-18T14:44:18.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T14:44:18.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T14:44:18.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo ip link delete ceph-brx 2024-03-18T14:44:18.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.368 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-18T14:44:18.384 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T14:44:18.385 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T14:44:18.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T14:44:18.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T14:44:18.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T14:44:18.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-03-18T14:44:18.617 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-18T14:44:18.627 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-18T14:44:18.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:44:18.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.729+0000 7fcabf827640 1 Processor -- start 2024-03-18T14:44:18.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.730+0000 7fcabf827640 1 -- start start 2024-03-18T14:44:18.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.730+0000 7fcabf827640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcab80b1740 0x7fcab80cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.730+0000 7fcabf827640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcab80b1b20 0x7fcab806b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.730+0000 7fcabf827640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.730+0000 7fcabf827640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fcab80ccd40 con 0x7fcab80b1b20 2024-03-18T14:44:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.730+0000 7fcabf827640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fcab80cd020 con 0x7fcab80b1740 2024-03-18T14:44:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.730+0000 7fcabf827640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcab80cd300 con 0x7fcab806b790 2024-03-18T14:44:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcab80b1b20 0x7fcab806b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcab80b1b20 0x7fcab806b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41428/0 (socket says 172.21.15.6:41428) 2024-03-18T14:44:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 -- 172.21.15.6:0/1226695491 learned_addr learned my addr 172.21.15.6:0/1226695491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabddc4640 1 --2- 172.21.15.6:0/1226695491 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabd5c3640 1 --2- 172.21.15.6:0/1226695491 >> v2:172.21.15.92:3300/0 conn(0x7fcab80b1740 0x7fcab80cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 -- 172.21.15.6:0/1226695491 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 msgr2=0x7fcab80caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:18.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 --2- 172.21.15.6:0/1226695491 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 -- 172.21.15.6:0/1226695491 >> v2:172.21.15.92:3300/0 conn(0x7fcab80b1740 msgr2=0x7fcab80cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:18.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 --2- 172.21.15.6:0/1226695491 >> v2:172.21.15.92:3300/0 conn(0x7fcab80b1740 0x7fcab80cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.731+0000 7fcabcdc2640 1 -- 172.21.15.6:0/1226695491 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab80cd5e0 con 0x7fcab80b1b20 2024-03-18T14:44:18.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabcdc2640 1 --2- 172.21.15.6:0/1226695491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1b20 0x7fcab806b250 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcab80d1aa0 tx=0x7fcaac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b8aa3af07fbbca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:18.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcab7fff640 1 -- 172.21.15.6:0/1226695491 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac034070 con 0x7fcab80b1b20 2024-03-18T14:44:18.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcab7fff640 1 -- 172.21.15.6:0/1226695491 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaac030860 con 0x7fcab80b1b20 2024-03-18T14:44:18.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcab7fff640 1 -- 172.21.15.6:0/1226695491 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac030c30 con 0x7fcab80b1b20 2024-03-18T14:44:18.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabf827640 1 -- 172.21.15.6:0/1226695491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1b20 msgr2=0x7fcab806b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabf827640 1 --2- 172.21.15.6:0/1226695491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1b20 0x7fcab806b250 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcab80d1aa0 tx=0x7fcaac02b9a0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabf827640 1 -- 172.21.15.6:0/1226695491 shutdown_connections 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabf827640 1 --2- 172.21.15.6:0/1226695491 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabf827640 1 --2- 172.21.15.6:0/1226695491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1b20 0x7fcab806b250 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabf827640 1 --2- 172.21.15.6:0/1226695491 >> v2:172.21.15.92:3300/0 conn(0x7fcab80b1740 0x7fcab80cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.732+0000 7fcabf827640 1 -- 172.21.15.6:0/1226695491 >> 172.21.15.6:0/1226695491 conn(0x7fcab80d09a0 msgr2=0x7fcab80d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.733+0000 7fcabf827640 1 -- 172.21.15.6:0/1226695491 shutdown_connections 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.733+0000 7fcabf827640 1 -- 172.21.15.6:0/1226695491 wait complete. 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 Processor -- start 2024-03-18T14:44:18.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 -- start start 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1740 0x7fcab81d5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcab80b1b20 0x7fcab80b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcab806bc00 con 0x7fcab80b1740 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcab80b1f90 con 0x7fcab80b1b20 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabf827640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fcab806bd80 con 0x7fcab806b790 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabcdc2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1740 0x7fcab81d5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabcdc2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1740 0x7fcab81d5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41434/0 (socket says 172.21.15.6:41434) 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.734+0000 7fcabcdc2640 1 -- 172.21.15.6:0/2006228346 learned_addr learned my addr 172.21.15.6:0/2006228346 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabd5c3640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabcdc2640 1 -- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 msgr2=0x7fcab80b6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabcdc2640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80b6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabcdc2640 1 -- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcab80b1b20 msgr2=0x7fcab80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabcdc2640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcab80b1b20 0x7fcab80b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabcdc2640 1 -- 172.21.15.6:0/2006228346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab81e0110 con 0x7fcab80b1740 2024-03-18T14:44:18.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabcdc2640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1740 0x7fcab81d5550 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fcaac02b9e0 tx=0x7fcaac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:18.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcab5ffb640 1 -- 172.21.15.6:0/2006228346 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac034040 con 0x7fcab80b1740 2024-03-18T14:44:18.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcab5ffb640 1 -- 172.21.15.6:0/2006228346 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaac030660 con 0x7fcab80b1740 2024-03-18T14:44:18.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab81e03b0 con 0x7fcab80b1740 2024-03-18T14:44:18.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcab5ffb640 1 -- 172.21.15.6:0/2006228346 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac039c10 con 0x7fcab80b1740 2024-03-18T14:44:18.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.735+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab81e0970 con 0x7fcab80b1740 2024-03-18T14:44:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.736+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab80d0d80 con 0x7fcab80b1740 2024-03-18T14:44:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.737+0000 7fcab5ffb640 1 -- 172.21.15.6:0/2006228346 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcaac037070 con 0x7fcab80b1740 2024-03-18T14:44:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.737+0000 7fcab5ffb640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcaa406bbd0 0x7fcaa406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.737+0000 7fcab5ffb640 1 -- 172.21.15.6:0/2006228346 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7fcaac0acad0 con 0x7fcab80b1740 2024-03-18T14:44:18.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.737+0000 7fcabd5c3640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcaa406bbd0 0x7fcaa406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:18.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.738+0000 7fcabd5c3640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcaa406bbd0 0x7fcaa406e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009a10 tx=0x7fcab0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:18.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.741+0000 7fcab5ffb640 1 -- 172.21.15.6:0/2006228346 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcaac030830 con 0x7fcab80b1740 2024-03-18T14:44:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.920+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcab81d5a90 con 0x7fcaa406bbd0 2024-03-18T14:44:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.922+0000 7fcab5ffb640 1 -- 172.21.15.6:0/2006228346 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7fcab81d5a90 con 0x7fcaa406bbd0 2024-03-18T14:44:18.923 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:44:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.923+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcaa406bbd0 msgr2=0x7fcaa406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.923+0000 7fcabf827640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcaa406bbd0 0x7fcaa406e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009a10 tx=0x7fcab0008040 comp rx=0 tx=0).stop 2024-03-18T14:44:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.923+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1740 msgr2=0x7fcab81d5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.923+0000 7fcabf827640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1740 0x7fcab81d5550 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fcaac02b9e0 tx=0x7fcaac0046c0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 shutdown_connections 2024-03-18T14:44:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fcaa406bbd0 0x7fcaa406e090 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcab80b1b20 0x7fcab80b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab80b1740 0x7fcab81d5550 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 --2- 172.21.15.6:0/2006228346 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcab806b790 0x7fcab80b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 >> 172.21.15.6:0/2006228346 conn(0x7fcab80d09a0 msgr2=0x7fcab806eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 shutdown_connections 2024-03-18T14:44:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:18.924+0000 7fcabf827640 1 -- 172.21.15.6:0/2006228346 wait complete. 2024-03-18T14:44:18.935 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-03-18T14:44:17.417070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222660,"kb_used_data":6496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752060,"statfs":{"total":773094113280,"available":772866109440,"internally_reserved":0,"allocated":6651904,"data_stored":3648120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001441"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":233,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973747+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:43:42.973747+0000","last_peered":"2024-03-18T14:43:42.973747+0000","last_clean":"2024-03-18T14:43:42.973747+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:43:42.973747+0000","last_undegraded":"2024-03-18T14:43:42.973747+0000","last_fullsized":"2024-03-18T14:43:42.973747+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":671744,"data_stored":304149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":894677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":696320,"data_stored":320447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":675840,"data_stored":304239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1294336,"data_stored":911397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:44:18.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:44:19.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.036+0000 7fa42f07e640 1 Processor -- start 2024-03-18T14:44:19.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.036+0000 7fa42f07e640 1 -- start start 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42f07e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa4280b17b0 0x7fa4280d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42f07e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4280cc990 0x7fa4280cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42f07e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4280b1c10 con 0x7fa4280d17f0 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42f07e640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa4280b1eb0 con 0x7fa4280cc990 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42f07e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fa4280b2190 con 0x7fa4280b17b0 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42d61b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4280cc990 0x7fa4280cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42ce1a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa4280d17f0 0x7fa4280d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55494/0 (socket says 172.21.15.6:55494) 2024-03-18T14:44:19.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42ce1a640 1 -- 172.21.15.6:0/69308989 learned_addr learned my addr 172.21.15.6:0/69308989 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:19.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa427fff640 1 --2- 172.21.15.6:0/69308989 >> v2:172.21.15.92:3300/0 conn(0x7fa4280b17b0 0x7fa4280d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42d61b640 1 -- 172.21.15.6:0/69308989 >> v2:172.21.15.92:3300/0 conn(0x7fa4280b17b0 msgr2=0x7fa4280d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:19.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42d61b640 1 --2- 172.21.15.6:0/69308989 >> v2:172.21.15.92:3300/0 conn(0x7fa4280b17b0 0x7fa4280d1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42d61b640 1 -- 172.21.15.6:0/69308989 >> v1:172.21.15.6:6789/0 conn(0x7fa4280d17f0 legacy=0x7fa4280d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa42d61b640 1 -- 172.21.15.6:0/69308989 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4280b2470 con 0x7fa4280cc990 2024-03-18T14:44:19.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.037+0000 7fa4277fe640 1 -- 172.21.15.6:0/69308989 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2734084919 0 0) 0x7fa4280b1c10 con 0x7fa4280d17f0 2024-03-18T14:44:19.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.038+0000 7fa42d61b640 1 --2- 172.21.15.6:0/69308989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280cc990 0x7fa4280cce20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa4280d1740 tx=0x7fa41802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d141e25533e052b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.038+0000 7fa4277fe640 1 -- 172.21.15.6:0/69308989 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418035020 con 0x7fa4280cc990 2024-03-18T14:44:19.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.038+0000 7fa4277fe640 1 -- 172.21.15.6:0/69308989 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418031830 con 0x7fa4280cc990 2024-03-18T14:44:19.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.038+0000 7fa4277fe640 1 -- 172.21.15.6:0/69308989 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418031c00 con 0x7fa4280cc990 2024-03-18T14:44:19.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 -- 172.21.15.6:0/69308989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280cc990 msgr2=0x7fa4280cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 --2- 172.21.15.6:0/69308989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280cc990 0x7fa4280cce20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa4280d1740 tx=0x7fa41802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 -- 172.21.15.6:0/69308989 shutdown_connections 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 --2- 172.21.15.6:0/69308989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280cc990 0x7fa4280cce20 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 --2- 172.21.15.6:0/69308989 >> v2:172.21.15.92:3300/0 conn(0x7fa4280b17b0 0x7fa4280d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 -- 172.21.15.6:0/69308989 >> 172.21.15.6:0/69308989 conn(0x7fa42808b130 msgr2=0x7fa42808b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 -- 172.21.15.6:0/69308989 shutdown_connections 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.039+0000 7fa42f07e640 1 -- 172.21.15.6:0/69308989 wait complete. 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.040+0000 7fa42f07e640 1 Processor -- start 2024-03-18T14:44:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.040+0000 7fa42f07e640 1 -- start start 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42f07e640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4280b17b0 0x7fa428150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42f07e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4280cc990 0x7fa42815e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42f07e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280d17f0 0x7fa4280ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42f07e640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa42815c080 con 0x7fa4280d17f0 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42f07e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa4280b2a30 con 0x7fa4280cc990 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42f07e640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fa4280d2110 con 0x7fa4280b17b0 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280d17f0 0x7fa4280ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280d17f0 0x7fa4280ce3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41442/0 (socket says 172.21.15.6:41442) 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 -- 172.21.15.6:0/3120608077 learned_addr learned my addr 172.21.15.6:0/3120608077 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa427fff640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4280cc990 0x7fa42815e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42ce1a640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4280b17b0 0x7fa428150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 -- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4280b17b0 msgr2=0x7fa428150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4280b17b0 0x7fa428150e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 -- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4280cc990 msgr2=0x7fa42815e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4280cc990 0x7fa42815e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42d61b640 1 -- 172.21.15.6:0/3120608077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4280c5a70 con 0x7fa4280d17f0 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa427fff640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4280cc990 0x7fa42815e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.041+0000 7fa42ce1a640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4280b17b0 0x7fa428150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.042+0000 7fa42d61b640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280d17f0 0x7fa4280ce3c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa41802bee0 tx=0x7fa4180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.042+0000 7fa4257fa640 1 -- 172.21.15.6:0/3120608077 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418035050 con 0x7fa4280d17f0 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.042+0000 7fa4257fa640 1 -- 172.21.15.6:0/3120608077 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418004230 con 0x7fa4280d17f0 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.042+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4280c5d50 con 0x7fa4280d17f0 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.042+0000 7fa4257fa640 1 -- 172.21.15.6:0/3120608077 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41803b780 con 0x7fa4280d17f0 2024-03-18T14:44:19.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.042+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4280c6310 con 0x7fa4280d17f0 2024-03-18T14:44:19.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.043+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42809fa80 con 0x7fa4280d17f0 2024-03-18T14:44:19.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.043+0000 7fa4257fa640 1 -- 172.21.15.6:0/3120608077 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa418038020 con 0x7fa4280d17f0 2024-03-18T14:44:19.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.043+0000 7fa4257fa640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bc00 0x7fa41406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.044+0000 7fa4257fa640 1 -- 172.21.15.6:0/3120608077 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7fa4180ac700 con 0x7fa4280d17f0 2024-03-18T14:44:19.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.044+0000 7fa42ce1a640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bc00 0x7fa41406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.044+0000 7fa42ce1a640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bc00 0x7fa41406e0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa420007fd0 tx=0x7fa420009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.048+0000 7fa4257fa640 1 -- 172.21.15.6:0/3120608077 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa418040070 con 0x7fa4280d17f0 2024-03-18T14:44:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.227+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa4280cf300 con 0x7fa41406bc00 2024-03-18T14:44:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.228+0000 7fa4257fa640 1 -- 172.21.15.6:0/3120608077 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7fa4280cf300 con 0x7fa41406bc00 2024-03-18T14:44:19.229 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:19.230 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:44:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bc00 msgr2=0x7fa41406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bc00 0x7fa41406e0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa420007fd0 tx=0x7fa420009040 comp rx=0 tx=0).stop 2024-03-18T14:44:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280d17f0 msgr2=0x7fa4280ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280d17f0 0x7fa4280ce3c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa41802bee0 tx=0x7fa4180046c0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 shutdown_connections 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fa41406bc00 0x7fa41406e0c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4280d17f0 0x7fa4280ce3c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4280cc990 0x7fa42815e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 --2- 172.21.15.6:0/3120608077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa4280b17b0 0x7fa428150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 >> 172.21.15.6:0/3120608077 conn(0x7fa42808b130 msgr2=0x7fa4280cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 shutdown_connections 2024-03-18T14:44:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.230+0000 7fa42f07e640 1 -- 172.21.15.6:0/3120608077 wait complete. 2024-03-18T14:44:19.242 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-03-18T14:44:17.417070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222660,"kb_used_data":6496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752060,"statfs":{"total":773094113280,"available":772866109440,"internally_reserved":0,"allocated":6651904,"data_stored":3648120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001441"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":233,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-18T14:43:42.973747+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:43:42.973747+0000","last_peered":"2024-03-18T14:43:42.973747+0000","last_clean":"2024-03-18T14:43:42.973747+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:43:42.973747+0000","last_undegraded":"2024-03-18T14:43:42.973747+0000","last_fullsized":"2024-03-18T14:43:42.973747+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":671744,"data_stored":304149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":894677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":696320,"data_stored":320447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":679936,"data_stored":304205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":675840,"data_stored":304239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1294336,"data_stored":911397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:44:19.242 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-18T14:44:19.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:44:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.342+0000 7f8343434640 1 Processor -- start 2024-03-18T14:44:19.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f8343434640 1 -- start start 2024-03-18T14:44:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f8343434640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f8343434640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f833c0b1b20 0x7f833c06b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f8343434640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f833c0ccd40 con 0x7f833c06b790 2024-03-18T14:44:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f8343434640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f833c0cd020 con 0x7f833c0b1b20 2024-03-18T14:44:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f8343434640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f833c0cd300 con 0x7f833c0b1740 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f83409cf640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f833c0b1b20 0x7f833c06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f83409cf640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f833c0b1b20 0x7f833c06b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41452/0 (socket says 172.21.15.6:41452) 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.343+0000 7f83409cf640 1 -- 172.21.15.6:0/3457503136 learned_addr learned my addr 172.21.15.6:0/3457503136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.344+0000 7f83411d0640 1 --2- 172.21.15.6:0/3457503136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.344+0000 7f83409cf640 1 -- 172.21.15.6:0/3457503136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 msgr2=0x7f833c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.344+0000 7f83409cf640 1 --2- 172.21.15.6:0/3457503136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.344+0000 7f83409cf640 1 -- 172.21.15.6:0/3457503136 >> v1:172.21.15.92:6789/0 conn(0x7f833c06b790 legacy=0x7f833c0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.344+0000 7f83409cf640 1 -- 172.21.15.6:0/3457503136 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833c0cd5e0 con 0x7f833c0b1b20 2024-03-18T14:44:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.344+0000 7f83411d0640 1 --2- 172.21.15.6:0/3457503136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.344+0000 7f83409cf640 1 --2- 172.21.15.6:0/3457503136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c0b1b20 0x7f833c06b250 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f832c009870 tx=0x7f832c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e881e7b1af27395 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8333fff640 1 -- 172.21.15.6:0/3457503136 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832c034070 con 0x7f833c0b1b20 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8333fff640 1 -- 172.21.15.6:0/3457503136 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f832c030840 con 0x7f833c0b1b20 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8333fff640 1 -- 172.21.15.6:0/3457503136 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832c030c10 con 0x7f833c0b1b20 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 -- 172.21.15.6:0/3457503136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c0b1b20 msgr2=0x7f833c06b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 --2- 172.21.15.6:0/3457503136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c0b1b20 0x7f833c06b250 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f832c009870 tx=0x7f832c02b980 comp rx=0 tx=0).stop 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 -- 172.21.15.6:0/3457503136 shutdown_connections 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 --2- 172.21.15.6:0/3457503136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c0b1b20 0x7f833c06b250 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 --2- 172.21.15.6:0/3457503136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 -- 172.21.15.6:0/3457503136 >> 172.21.15.6:0/3457503136 conn(0x7f833c0d09a0 msgr2=0x7f833c0d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 -- 172.21.15.6:0/3457503136 shutdown_connections 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.345+0000 7f8343434640 1 -- 172.21.15.6:0/3457503136 wait complete. 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.346+0000 7f8343434640 1 Processor -- start 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f8343434640 1 -- start start 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f8343434640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c06b790 0x7f833c1f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f8343434640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f8343434640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c0b1b20 0x7f833c0b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f8343434640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f833c0b1f60 con 0x7f833c06b790 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f8343434640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f833c06bc00 con 0x7f833c0b1b20 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f8343434640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f833c0b5150 con 0x7f833c0b1740 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f83411d0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c06b790 0x7f833c1f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f83411d0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c06b790 0x7f833c1f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41466/0 (socket says 172.21.15.6:41466) 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.347+0000 7f83411d0640 1 -- 172.21.15.6:0/97058746 learned_addr learned my addr 172.21.15.6:0/97058746 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83409cf640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83419d1640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c0b1b20 0x7f833c0b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83411d0640 1 -- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 msgr2=0x7f833c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83411d0640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83411d0640 1 -- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c0b1b20 msgr2=0x7f833c0b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83411d0640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c0b1b20 0x7f833c0b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83411d0640 1 -- 172.21.15.6:0/97058746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833c1f4ce0 con 0x7f833c06b790 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83419d1640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c0b1b20 0x7f833c0b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f83411d0640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c06b790 0x7f833c1f8540 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f833800c820 tx=0x7f833800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f8331ffb640 1 -- 172.21.15.6:0/97058746 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338015070 con 0x7f833c06b790 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f8331ffb640 1 -- 172.21.15.6:0/97058746 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8338007a30 con 0x7f833c06b790 2024-03-18T14:44:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f833c1f4fc0 con 0x7f833c06b790 2024-03-18T14:44:19.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f8331ffb640 1 -- 172.21.15.6:0/97058746 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338007de0 con 0x7f833c06b790 2024-03-18T14:44:19.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.348+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f833c1f52a0 con 0x7f833c06b790 2024-03-18T14:44:19.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.349+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f833c067400 con 0x7f833c06b790 2024-03-18T14:44:19.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.349+0000 7f8331ffb640 1 -- 172.21.15.6:0/97058746 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8338018070 con 0x7f833c06b790 2024-03-18T14:44:19.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.350+0000 7f8331ffb640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f832006bbd0 0x7f832006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.350+0000 7f8331ffb640 1 -- 172.21.15.6:0/97058746 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f833808a850 con 0x7f833c06b790 2024-03-18T14:44:19.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.350+0000 7f83409cf640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f832006bbd0 0x7f832006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.350+0000 7f83409cf640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f832006bbd0 0x7f832006e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f832c036000 tx=0x7f832c03b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.354+0000 7f8331ffb640 1 -- 172.21.15.6:0/97058746 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f833c067400 con 0x7f833c06b790 2024-03-18T14:44:19.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.534+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f833c04f620 con 0x7f832006bbd0 2024-03-18T14:44:19.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.535+0000 7f8331ffb640 1 -- 172.21.15.6:0/97058746 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13487 (secure 0 0 0) 0x7f833c04f620 con 0x7f832006bbd0 2024-03-18T14:44:19.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:19.537 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:44:19.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f832006bbd0 msgr2=0x7f832006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f832006bbd0 0x7f832006e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f832c036000 tx=0x7f832c03b040 comp rx=0 tx=0).stop 2024-03-18T14:44:19.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c06b790 msgr2=0x7f833c1f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c06b790 0x7f833c1f8540 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f833800c820 tx=0x7f833800cce0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 shutdown_connections 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f832006bbd0 0x7f832006e090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c0b1b20 0x7f833c0b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b1740 0x7f833c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 --2- 172.21.15.6:0/97058746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f833c06b790 0x7f833c1f8540 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 >> 172.21.15.6:0/97058746 conn(0x7f833c0d09a0 msgr2=0x7f833c070250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:19.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 shutdown_connections 2024-03-18T14:44:19.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.538+0000 7f8343434640 1 -- 172.21.15.6:0/97058746 wait complete. 2024-03-18T14:44:19.549 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-03-18T14:44:19.417428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222944,"kb_used_data":6760,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751776,"statfs":{"total":773094113280,"available":772865818624,"internally_reserved":0,"allocated":6922240,"data_stored":3764528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17096,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-70,"num_write_kb":-41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001418"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-18T14:44:17.198468+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:44:17.198468+0000","last_peered":"2024-03-18T14:44:17.198468+0000","last_clean":"2024-03-18T14:44:17.198468+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:44:17.198468+0000","last_undegraded":"2024-03-18T14:44:17.198468+0000","last_fullsized":"2024-03-18T14:44:17.198468+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":913342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":913342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:44:19.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T14:44:19.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.651+0000 7f1f29a8c640 1 Processor -- start 2024-03-18T14:44:19.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.651+0000 7f1f29a8c640 1 -- start start 2024-03-18T14:44:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f29a8c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1f240cd870 0x7f1f240ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f29a8c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f240c9080 0x7f1f240ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f29a8c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f24071e10 con 0x7f1f24147960 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f29a8c640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1f24147d40 con 0x7f1f240c9080 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f29a8c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1f24072a20 con 0x7f1f240cd870 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f23fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f240c9080 0x7f1f240ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f23fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f240c9080 0x7f1f240ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41478/0 (socket says 172.21.15.6:41478) 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f23fff640 1 -- 172.21.15.6:0/1493966318 learned_addr learned my addr 172.21.15.6:0/1493966318 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f22ffd640 1 --2- 172.21.15.6:0/1493966318 >> v2:172.21.15.92:3300/0 conn(0x7f1f240cd870 0x7f1f240ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f23fff640 1 -- 172.21.15.6:0/1493966318 >> v2:172.21.15.92:3300/0 conn(0x7f1f240cd870 msgr2=0x7f1f240ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f23fff640 1 --2- 172.21.15.6:0/1493966318 >> v2:172.21.15.92:3300/0 conn(0x7f1f240cd870 0x7f1f240ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f23fff640 1 -- 172.21.15.6:0/1493966318 >> v1:172.21.15.92:6789/0 conn(0x7f1f24147960 legacy=0x7f1f240ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:19.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.652+0000 7f1f23fff640 1 -- 172.21.15.6:0/1493966318 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f240720f0 con 0x7f1f240c9080 2024-03-18T14:44:19.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.653+0000 7f1f23fff640 1 --2- 172.21.15.6:0/1493966318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f240c9080 0x7f1f240ca830 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1f240cbd70 tx=0x7f1f1c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f8d5940d38d40c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.653+0000 7f1f227fc640 1 -- 172.21.15.6:0/1493966318 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c02c9f0 con 0x7f1f240c9080 2024-03-18T14:44:19.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.653+0000 7f1f227fc640 1 -- 172.21.15.6:0/1493966318 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f1c02cb90 con 0x7f1f240c9080 2024-03-18T14:44:19.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.653+0000 7f1f227fc640 1 -- 172.21.15.6:0/1493966318 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c0343f0 con 0x7f1f240c9080 2024-03-18T14:44:19.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 -- 172.21.15.6:0/1493966318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f240c9080 msgr2=0x7f1f240ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/1493966318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f240c9080 0x7f1f240ca830 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1f240cbd70 tx=0x7f1f1c02be50 comp rx=0 tx=0).stop 2024-03-18T14:44:19.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 -- 172.21.15.6:0/1493966318 shutdown_connections 2024-03-18T14:44:19.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/1493966318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f240c9080 0x7f1f240ca830 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/1493966318 >> v2:172.21.15.92:3300/0 conn(0x7f1f240cd870 0x7f1f240ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 -- 172.21.15.6:0/1493966318 >> 172.21.15.6:0/1493966318 conn(0x7f1f240cc7b0 msgr2=0x7f1f240ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:19.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 -- 172.21.15.6:0/1493966318 shutdown_connections 2024-03-18T14:44:19.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.654+0000 7f1f29a8c640 1 -- 172.21.15.6:0/1493966318 wait complete. 2024-03-18T14:44:19.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.655+0000 7f1f29a8c640 1 Processor -- start 2024-03-18T14:44:19.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.655+0000 7f1f29a8c640 1 -- start start 2024-03-18T14:44:19.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f29a8c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f240c9080 0x7f1f2413ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f29a8c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f240cd870 0x7f1f24140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f29a8c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f24147960 0x7f1f24141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f29a8c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1f240cd4b0 con 0x7f1f24147960 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f29a8c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1f240726b0 con 0x7f1f240cd870 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f29a8c640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1f240cd630 con 0x7f1f240c9080 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f24147960 0x7f1f24141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f24147960 0x7f1f24141ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41488/0 (socket says 172.21.15.6:41488) 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 -- 172.21.15.6:0/3231671376 learned_addr learned my addr 172.21.15.6:0/3231671376 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f237fe640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f240c9080 0x7f1f2413ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 -- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f240c9080 msgr2=0x7f1f2413ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f22ffd640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f240cd870 0x7f1f24140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f240c9080 0x7f1f2413ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 -- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f240cd870 msgr2=0x7f1f24140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f240cd870 0x7f1f24140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 -- 172.21.15.6:0/3231671376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2413f7e0 con 0x7f1f24147960 2024-03-18T14:44:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.656+0000 7f1f23fff640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f24147960 0x7f1f24141ac0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c009f40 tx=0x7f1f1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.657+0000 7f1f0bfff640 1 -- 172.21.15.6:0/3231671376 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c043020 con 0x7f1f24147960 2024-03-18T14:44:19.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.657+0000 7f1f0bfff640 1 -- 172.21.15.6:0/3231671376 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f1c03e450 con 0x7f1f24147960 2024-03-18T14:44:19.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.657+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f2413fac0 con 0x7f1f24147960 2024-03-18T14:44:19.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.657+0000 7f1f0bfff640 1 -- 172.21.15.6:0/3231671376 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c009510 con 0x7f1f24147960 2024-03-18T14:44:19.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.657+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f24140360 con 0x7f1f24147960 2024-03-18T14:44:19.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.658+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f24073700 con 0x7f1f24147960 2024-03-18T14:44:19.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.658+0000 7f1f0bfff640 1 -- 172.21.15.6:0/3231671376 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1f1c046020 con 0x7f1f24147960 2024-03-18T14:44:19.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.658+0000 7f1f0bfff640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1f1006bc00 0x7f1f1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:19.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.659+0000 7f1f0bfff640 1 -- 172.21.15.6:0/3231671376 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f1f1c0ace10 con 0x7f1f24147960 2024-03-18T14:44:19.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.659+0000 7f1f237fe640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1f1006bc00 0x7f1f1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:19.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.659+0000 7f1f237fe640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1f1006bc00 0x7f1f1006e0c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1f14003d90 tx=0x7f1f14009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:19.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.663+0000 7f1f0bfff640 1 -- 172.21.15.6:0/3231671376 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1f1c04d070 con 0x7f1f24147960 2024-03-18T14:44:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.846+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f2406c0c0 con 0x7f1f24147960 2024-03-18T14:44:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.846+0000 7f1f0bfff640 1 -- 172.21.15.6:0/3231671376 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+13195 (secure 0 0 0) 0x7f1f1c07bb00 con 0x7f1f24147960 2024-03-18T14:44:19.848 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":59,"fsid":"a75a8f99-9e80-46f7-9da4-368af92e8063","created":"2024-03-18T14:41:11.150867+0000","modified":"2024-03-18T14:44:16.553019+0000","last_up_change":"2024-03-18T14:41:22.399350+0000","last_in_change":"2024-03-18T14:41:13.405413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T14:41:23.427896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"297ee578-07f6-413d-94d4-1feabcab858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6825","nonce":3434665479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6827","nonce":3434665479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6831","nonce":3434665479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3434665479},{"type":"v1","addr":"172.21.15.6:6829","nonce":3434665479}]},"public_addr":"172.21.15.6:6825/3434665479","cluster_addr":"172.21.15.6:6827/3434665479","heartbeat_back_addr":"172.21.15.6:6831/3434665479","heartbeat_front_addr":"172.21.15.6:6829/3434665479","state":["exists","up"]},{"osd":1,"uuid":"0517bd2c-8330-4ffb-9920-e4d01560c349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6810","nonce":2441154404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6814","nonce":2441154404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6819","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6820","nonce":2441154404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2441154404},{"type":"v1","addr":"172.21.15.6:6817","nonce":2441154404}]},"public_addr":"172.21.15.6:6810/2441154404","cluster_addr":"172.21.15.6:6814/2441154404","heartbeat_back_addr":"172.21.15.6:6820/2441154404","heartbeat_front_addr":"172.21.15.6:6817/2441154404","state":["exists","up"]},{"osd":2,"uuid":"b840e6ce-6fc9-4428-a13c-e45ee2431bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6801","nonce":3396973445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6803","nonce":3396973445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6807","nonce":3396973445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3396973445},{"type":"v1","addr":"172.21.15.6:6805","nonce":3396973445}]},"public_addr":"172.21.15.6:6801/3396973445","cluster_addr":"172.21.15.6:6803/3396973445","heartbeat_back_addr":"172.21.15.6:6807/3396973445","heartbeat_front_addr":"172.21.15.6:6805/3396973445","state":["exists","up"]},{"osd":3,"uuid":"2cf1864e-77e7-4115-b871-12395a882274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6811","nonce":536857262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6816","nonce":536857262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6823","nonce":536857262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":536857262},{"type":"v1","addr":"172.21.15.6:6821","nonce":536857262}]},"public_addr":"172.21.15.6:6811/536857262","cluster_addr":"172.21.15.6:6816/536857262","heartbeat_back_addr":"172.21.15.6:6823/536857262","heartbeat_front_addr":"172.21.15.6:6821/536857262","state":["exists","up"]},{"osd":4,"uuid":"ad5dd4ac-44f8-4929-807d-244709ff1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6817","nonce":3636494559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6819","nonce":3636494559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6823","nonce":3636494559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3636494559},{"type":"v1","addr":"172.21.15.92:6821","nonce":3636494559}]},"public_addr":"172.21.15.92:6817/3636494559","cluster_addr":"172.21.15.92:6819/3636494559","heartbeat_back_addr":"172.21.15.92:6823/3636494559","heartbeat_front_addr":"172.21.15.92:6821/3636494559","state":["exists","up"]},{"osd":5,"uuid":"7390e35b-e1e4-4b0d-87da-923c0952b58c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6809","nonce":4136952425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6811","nonce":4136952425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6815","nonce":4136952425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":4136952425},{"type":"v1","addr":"172.21.15.92:6813","nonce":4136952425}]},"public_addr":"172.21.15.92:6809/4136952425","cluster_addr":"172.21.15.92:6811/4136952425","heartbeat_back_addr":"172.21.15.92:6815/4136952425","heartbeat_front_addr":"172.21.15.92:6813/4136952425","state":["exists","up"]},{"osd":6,"uuid":"01baa0b2-56c2-436f-893b-2282f86b7e05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6801","nonce":1041453586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6803","nonce":1041453586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6807","nonce":1041453586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1041453586},{"type":"v1","addr":"172.21.15.92:6805","nonce":1041453586}]},"public_addr":"172.21.15.92:6801/1041453586","cluster_addr":"172.21.15.92:6803/1041453586","heartbeat_back_addr":"172.21.15.92:6807/1041453586","heartbeat_front_addr":"172.21.15.92:6805/1041453586","state":["exists","up"]},{"osd":7,"uuid":"d49cb14a-4510-423d-86a8-cb12fff9c960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6825","nonce":1103567419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6827","nonce":1103567419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6831","nonce":1103567419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1103567419},{"type":"v1","addr":"172.21.15.92:6829","nonce":1103567419}]},"public_addr":"172.21.15.92:6825/1103567419","cluster_addr":"172.21.15.92:6827/1103567419","heartbeat_back_addr":"172.21.15.92:6831/1103567419","heartbeat_front_addr":"172.21.15.92:6829/1103567419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.545476+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.506061+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.607559+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.579231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.606932+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.596640+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T14:41:18.588750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6834/3070832840":"2024-03-19T14:44:13.544623+0000","172.21.15.6:6832/3070832840":"2024-03-19T14:44:13.544623+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T14:44:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1f1006bc00 msgr2=0x7f1f1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1f1006bc00 0x7f1f1006e0c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1f14003d90 tx=0x7f1f14009040 comp rx=0 tx=0).stop 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f24147960 msgr2=0x7f1f24141ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f24147960 0x7f1f24141ac0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c009f40 tx=0x7f1f1c004a40 comp rx=0 tx=0).stop 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 shutdown_connections 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1f1006bc00 0x7f1f1006e0c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f24147960 0x7f1f24141ac0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f240cd870 0x7f1f24140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 --2- 172.21.15.6:0/3231671376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f240c9080 0x7f1f2413ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 >> 172.21.15.6:0/3231671376 conn(0x7f1f240cc7b0 msgr2=0x7f1f240ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 shutdown_connections 2024-03-18T14:44:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:19.849+0000 7f1f29a8c640 1 -- 172.21.15.6:0/3231671376 wait complete. 2024-03-18T14:44:20.862 INFO:tasks.ceph:Scrubbing osd.0 2024-03-18T14:44:20.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.980+0000 7f0f470ff640 1 Processor -- start 2024-03-18T14:44:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.980+0000 7f0f470ff640 1 -- start start 2024-03-18T14:44:20.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.981+0000 7f0f470ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0f4006ecb0 0x7f0f40070340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:20.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.981+0000 7f0f470ff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0f40070880 0x7f0f4015d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:20.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f470ff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f400d0170 con 0x7f0f4006b1c0 2024-03-18T14:44:20.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f470ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0f400cdc20 con 0x7f0f4006ecb0 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f470ff640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f0f400d0450 con 0x7f0f40070880 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f3ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0f4006ecb0 0x7f0f40070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f3ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0f4006ecb0 0x7f0f40070340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41494/0 (socket says 172.21.15.6:41494) 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f3ffff640 1 -- 172.21.15.6:0/1986487968 learned_addr learned my addr 172.21.15.6:0/1986487968 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f4569c640 1 --2- 172.21.15.6:0/1986487968 >> v2:172.21.15.92:3300/0 conn(0x7f0f40070880 0x7f0f4015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f3ffff640 1 -- 172.21.15.6:0/1986487968 >> v2:172.21.15.92:3300/0 conn(0x7f0f40070880 msgr2=0x7f0f4015d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f3ffff640 1 --2- 172.21.15.6:0/1986487968 >> v2:172.21.15.92:3300/0 conn(0x7f0f40070880 0x7f0f4015d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f3ffff640 1 -- 172.21.15.6:0/1986487968 >> v1:172.21.15.92:6789/0 conn(0x7f0f4006b1c0 legacy=0x7f0f4006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.982+0000 7f0f3ffff640 1 -- 172.21.15.6:0/1986487968 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f400cd8b0 con 0x7f0f4006ecb0 2024-03-18T14:44:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.983+0000 7f0f3ffff640 1 --2- 172.21.15.6:0/1986487968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006ecb0 0x7f0f40070340 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009870 tx=0x7f0f2c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd1edf97776db67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.983+0000 7f0f3f7fe640 1 -- 172.21.15.6:0/1986487968 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c02e070 con 0x7f0f4006ecb0 2024-03-18T14:44:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.983+0000 7f0f3f7fe640 1 -- 172.21.15.6:0/1986487968 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f2c037070 con 0x7f0f4006ecb0 2024-03-18T14:44:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.983+0000 7f0f3f7fe640 1 -- 172.21.15.6:0/1986487968 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c031a00 con 0x7f0f4006ecb0 2024-03-18T14:44:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 -- 172.21.15.6:0/1986487968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006ecb0 msgr2=0x7f0f40070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 --2- 172.21.15.6:0/1986487968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006ecb0 0x7f0f40070340 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009870 tx=0x7f0f2c02b880 comp rx=0 tx=0).stop 2024-03-18T14:44:20.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 -- 172.21.15.6:0/1986487968 shutdown_connections 2024-03-18T14:44:20.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 --2- 172.21.15.6:0/1986487968 >> v2:172.21.15.92:3300/0 conn(0x7f0f40070880 0x7f0f4015d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:20.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 --2- 172.21.15.6:0/1986487968 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006ecb0 0x7f0f40070340 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:20.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 -- 172.21.15.6:0/1986487968 >> 172.21.15.6:0/1986487968 conn(0x7f0f400d09f0 msgr2=0x7f0f400d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:20.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 -- 172.21.15.6:0/1986487968 shutdown_connections 2024-03-18T14:44:20.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.984+0000 7f0f470ff640 1 -- 172.21.15.6:0/1986487968 wait complete. 2024-03-18T14:44:20.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.985+0000 7f0f470ff640 1 Processor -- start 2024-03-18T14:44:20.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.985+0000 7f0f470ff640 1 -- start start 2024-03-18T14:44:20.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f470ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006b1c0 0x7f0f400b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:20.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f470ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f4006ecb0 0x7f0f401f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f470ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f40070880 0x7f0f401f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f470ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0f400ce0b0 con 0x7f0f4006b1c0 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f470ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0f400d0730 con 0x7f0f40070880 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f470ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0f400ce230 con 0x7f0f4006ecb0 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f44e9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006b1c0 0x7f0f400b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f44e9b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006b1c0 0x7f0f400b2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41498/0 (socket says 172.21.15.6:41498) 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.986+0000 7f0f44e9b640 1 -- 172.21.15.6:0/1686930691 learned_addr learned my addr 172.21.15.6:0/1686930691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:20.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f3ffff640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f4006ecb0 0x7f0f401f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f4569c640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f40070880 0x7f0f401f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f44e9b640 1 -- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f4006ecb0 msgr2=0x7f0f401f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f44e9b640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f4006ecb0 0x7f0f401f4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f44e9b640 1 -- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f40070880 msgr2=0x7f0f401f48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f44e9b640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f40070880 0x7f0f401f48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f44e9b640 1 -- 172.21.15.6:0/1686930691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f401e5170 con 0x7f0f4006b1c0 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f44e9b640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006b1c0 0x7f0f400b2d70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400c3c0 tx=0x7f0f3400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f340052d0 con 0x7f0f4006b1c0 2024-03-18T14:44:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f34005490 con 0x7f0f4006b1c0 2024-03-18T14:44:20.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f34007260 con 0x7f0f4006b1c0 2024-03-18T14:44:20.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f400b5e30 con 0x7f0f4006b1c0 2024-03-18T14:44:20.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.987+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f400b3770 con 0x7f0f4006b1c0 2024-03-18T14:44:20.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.988+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f400b4110 con 0x7f0f4006b1c0 2024-03-18T14:44:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0f34007400 con 0x7f0f4006b1c0 2024-03-18T14:44:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f3d7fa640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0f2406bc00 0x7f0f2406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f0f34089ca0 con 0x7f0f4006b1c0 2024-03-18T14:44:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f3d7fa640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7f0f240720a0 0x7f0f240744e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 --> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f34010e50 con 0x7f0f240720a0 2024-03-18T14:44:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f0f3408c3a0 con 0x7f0f4006b1c0 2024-03-18T14:44:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f3ffff640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0f2406bc00 0x7f0f2406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.989+0000 7f0f4569c640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7f0f240720a0 0x7f0f240744e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.990+0000 7f0f4569c640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7f0f240720a0 0x7f0f240744e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.990+0000 7f0f3ffff640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0f2406bc00 0x7f0f2406e0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c036000 tx=0x7f0f2c03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:20.991+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== osd.0 v2:172.21.15.6:6824/3434665479 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f34010e50 con 0x7f0f240720a0 2024-03-18T14:44:21.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.004+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 --> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0f400b4e60 con 0x7f0f240720a0 2024-03-18T14:44:21.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.004+0000 7f0f3d7fa640 1 -- 172.21.15.6:0/1686930691 <== osd.0 v2:172.21.15.6:6824/3434665479 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0f400b4e60 con 0x7f0f240720a0 2024-03-18T14:44:21.006 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:21.006 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:21.006 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:21.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7f0f240720a0 msgr2=0x7f0f240744e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:6824/3434665479,v1:172.21.15.6:6825/3434665479] conn(0x7f0f240720a0 0x7f0f240744e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0f2406bc00 msgr2=0x7f0f2406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f0f2406bc00 0x7f0f2406e0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c036000 tx=0x7f0f2c03d040 comp rx=0 tx=0).stop 2024-03-18T14:44:21.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006b1c0 msgr2=0x7f0f400b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 --2- 172.21.15.6:0/1686930691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f4006b1c0 0x7f0f400b2d70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400c3c0 tx=0x7f0f3400c880 comp rx=0 tx=0).stop 2024-03-18T14:44:21.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f4569c640 1 -- 172.21.15.6:0/1686930691 reap_dead start 2024-03-18T14:44:21.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 shutdown_connections 2024-03-18T14:44:21.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 >> 172.21.15.6:0/1686930691 conn(0x7f0f400d09f0 msgr2=0x7f0f4006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 shutdown_connections 2024-03-18T14:44:21.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.005+0000 7f0f470ff640 1 -- 172.21.15.6:0/1686930691 wait complete. 2024-03-18T14:44:21.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-18T14:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.119+0000 7f1af88c0640 1 Processor -- start 2024-03-18T14:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.119+0000 7f1af88c0640 1 -- start start 2024-03-18T14:44:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af88c0640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1af406c7a0 0x7f1af40d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af88c0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1af406cb80 0x7f1af40b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af88c0640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af40d1f30 con 0x7f1af40b6df0 2024-03-18T14:44:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af88c0640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1af40d2210 con 0x7f1af406c7a0 2024-03-18T14:44:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af88c0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f1af40d24f0 con 0x7f1af406cb80 2024-03-18T14:44:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af259c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1af406c7a0 0x7f1af40d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af259c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1af406c7a0 0x7f1af40d2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41502/0 (socket says 172.21.15.6:41502) 2024-03-18T14:44:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af259c640 1 -- 172.21.15.6:0/3576790941 learned_addr learned my addr 172.21.15.6:0/3576790941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1aebfff640 1 --2- 172.21.15.6:0/3576790941 >> v2:172.21.15.92:3300/0 conn(0x7f1af406cb80 0x7f1af40b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af259c640 1 -- 172.21.15.6:0/3576790941 >> v2:172.21.15.92:3300/0 conn(0x7f1af406cb80 msgr2=0x7f1af40b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af259c640 1 --2- 172.21.15.6:0/3576790941 >> v2:172.21.15.92:3300/0 conn(0x7f1af406cb80 0x7f1af40b95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af259c640 1 -- 172.21.15.6:0/3576790941 >> v1:172.21.15.92:6789/0 conn(0x7f1af40b6df0 legacy=0x7f1af40b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.120+0000 7f1af259c640 1 -- 172.21.15.6:0/3576790941 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af40d27d0 con 0x7f1af406c7a0 2024-03-18T14:44:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.121+0000 7f1af259c640 1 --2- 172.21.15.6:0/3576790941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af40d2fd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1af40d1650 tx=0x7f1ae402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb9be14f9f86f6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.121+0000 7f1af1d9b640 1 -- 172.21.15.6:0/3576790941 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae402f850 con 0x7f1af406c7a0 2024-03-18T14:44:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.121+0000 7f1af1d9b640 1 -- 172.21.15.6:0/3576790941 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae402f9f0 con 0x7f1af406c7a0 2024-03-18T14:44:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.121+0000 7f1af1d9b640 1 -- 172.21.15.6:0/3576790941 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae402fdc0 con 0x7f1af406c7a0 2024-03-18T14:44:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 -- 172.21.15.6:0/3576790941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 msgr2=0x7f1af40d2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 --2- 172.21.15.6:0/3576790941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af40d2fd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1af40d1650 tx=0x7f1ae402ba30 comp rx=0 tx=0).stop 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 -- 172.21.15.6:0/3576790941 shutdown_connections 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 --2- 172.21.15.6:0/3576790941 >> v2:172.21.15.92:3300/0 conn(0x7f1af406cb80 0x7f1af40b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 --2- 172.21.15.6:0/3576790941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af40d2fd0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 -- 172.21.15.6:0/3576790941 >> 172.21.15.6:0/3576790941 conn(0x7f1af4072ce0 msgr2=0x7f1af40730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 -- 172.21.15.6:0/3576790941 shutdown_connections 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.122+0000 7f1af88c0640 1 -- 172.21.15.6:0/3576790941 wait complete. 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.123+0000 7f1af88c0640 1 Processor -- start 2024-03-18T14:44:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.123+0000 7f1af88c0640 1 -- start start 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af88c0640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af4145f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af88c0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af406cb80 0x7f1af40b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af88c0640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1af40b6df0 0x7f1af4148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af88c0640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1af40b9b20 con 0x7f1af406c7a0 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af88c0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1af406cfe0 con 0x7f1af406cb80 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af88c0640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f1af40b9ca0 con 0x7f1af40b6df0 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af4145f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af4145f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41516/0 (socket says 172.21.15.6:41516) 2024-03-18T14:44:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 -- 172.21.15.6:0/2430447089 learned_addr learned my addr 172.21.15.6:0/2430447089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af2d9d640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1af40b6df0 0x7f1af4148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1aebfff640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af406cb80 0x7f1af40b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 -- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1af40b6df0 msgr2=0x7f1af4148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1af40b6df0 0x7f1af4148750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 -- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af406cb80 msgr2=0x7f1af40b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af406cb80 0x7f1af40b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1af259c640 1 -- 172.21.15.6:0/2430447089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af40b4a10 con 0x7f1af406c7a0 2024-03-18T14:44:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.124+0000 7f1aebfff640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af406cb80 0x7f1af40b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.125+0000 7f1af259c640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af4145f70 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1ae402ba70 tx=0x7f1ae4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.125+0000 7f1aeaffd640 1 -- 172.21.15.6:0/2430447089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4037d10 con 0x7f1af406c7a0 2024-03-18T14:44:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.125+0000 7f1aeaffd640 1 -- 172.21.15.6:0/2430447089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae4046070 con 0x7f1af406c7a0 2024-03-18T14:44:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.125+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af40b4cf0 con 0x7f1af406c7a0 2024-03-18T14:44:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.125+0000 7f1aeaffd640 1 -- 172.21.15.6:0/2430447089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4041640 con 0x7f1af406c7a0 2024-03-18T14:44:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.125+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af4146760 con 0x7f1af406c7a0 2024-03-18T14:44:21.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.126+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af404f5b0 con 0x7f1af406c7a0 2024-03-18T14:44:21.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.126+0000 7f1aeaffd640 1 -- 172.21.15.6:0/2430447089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1ae4040070 con 0x7f1af406c7a0 2024-03-18T14:44:21.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.127+0000 7f1aeaffd640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ad406bbd0 0x7f1ad406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.127+0000 7f1aeaffd640 1 -- 172.21.15.6:0/2430447089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f1ae40acf60 con 0x7f1af406c7a0 2024-03-18T14:44:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.127+0000 7f1aebfff640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ad406bbd0 0x7f1ad406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.128+0000 7f1aebfff640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ad406bbd0 0x7f1ad406e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1af40b5c90 tx=0x7f1adc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.131+0000 7f1aeaffd640 1 -- 172.21.15.6:0/2430447089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1ae4033030 con 0x7f1af406c7a0 2024-03-18T14:44:21.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.315+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f1af40ca190 con 0x7f1ad406bbd0 2024-03-18T14:44:21.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.315+0000 7f1aeaffd640 1 -- 172.21.15.6:0/2430447089 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1af40ca190 con 0x7f1ad406bbd0 2024-03-18T14:44:21.316 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-03-18T14:44:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ad406bbd0 msgr2=0x7f1ad406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ad406bbd0 0x7f1ad406e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1af40b5c90 tx=0x7f1adc008040 comp rx=0 tx=0).stop 2024-03-18T14:44:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 msgr2=0x7f1af4145f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af4145f70 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1ae402ba70 tx=0x7f1ae4004730 comp rx=0 tx=0).stop 2024-03-18T14:44:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 shutdown_connections 2024-03-18T14:44:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f1ad406bbd0 0x7f1ad406e090 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1af40b6df0 0x7f1af4148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af406cb80 0x7f1af40b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 --2- 172.21.15.6:0/2430447089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1af406c7a0 0x7f1af4145f70 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 >> 172.21.15.6:0/2430447089 conn(0x7f1af4072ce0 msgr2=0x7f1af40b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 shutdown_connections 2024-03-18T14:44:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.317+0000 7f1af88c0640 1 -- 172.21.15.6:0/2430447089 wait complete. 2024-03-18T14:44:21.329 INFO:tasks.ceph:Scrubbing osd.1 2024-03-18T14:44:21.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:21.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.435+0000 7f519d861640 1 Processor -- start 2024-03-18T14:44:21.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.435+0000 7f519d861640 1 -- start start 2024-03-18T14:44:21.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f519d861640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f51981424f0 0x7f51980718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f519d861640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5198071e30 0x7f51980ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f519d861640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5198072390 con 0x7f51980cd2b0 2024-03-18T14:44:21.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f519d861640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f51980cd720 con 0x7f51981424f0 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f519d861640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f51981428d0 con 0x7f5198071e30 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f51977fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f51980cd2b0 0x7f519815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55500/0 (socket says 172.21.15.6:55500) 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f51977fe640 1 -- 172.21.15.6:0/2242961142 learned_addr learned my addr 172.21.15.6:0/2242961142 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919255909 0 0) 0x7f5198072390 con 0x7f51980cd2b0 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5180003520 con 0x7f51980cd2b0 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5196ffd640 1 --2- 172.21.15.6:0/2242961142 >> v2:172.21.15.92:3300/0 conn(0x7f51981424f0 0x7f51980718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f51967fc640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5198071e30 0x7f51980ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 90634669 0 0) 0x7f5180003520 con 0x7f51980cd2b0 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5198071e30 msgr2=0x7f51980ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5198071e30 0x7f51980ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 >> v2:172.21.15.92:3300/0 conn(0x7f51981424f0 msgr2=0x7f51980718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 --2- 172.21.15.6:0/2242961142 >> v2:172.21.15.92:3300/0 conn(0x7f51981424f0 0x7f51980718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5198072670 con 0x7f51980cd2b0 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.436+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f51900019f0 con 0x7f51980cd2b0 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 >> v1:172.21.15.6:6789/0 conn(0x7f51980cd2b0 legacy=0x7f519815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5195ffb640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51800039b0 0x7f5180003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5195ffb640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51800043a0 0x7f5180004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5195ffb640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5180004ef0 0x7f518000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f51980cd720 con 0x7f5180004ef0 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f51981428d0 con 0x7f51800043a0 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f518000bf00 con 0x7f51800039b0 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51967fc640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5198071e30 0x7f51980ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51977fe640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5180004ef0 0x7f518000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f5196ffd640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51800039b0 0x7f5180003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51967fc640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51800043a0 0x7f5180004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51977fe640 1 -- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51800039b0 msgr2=0x7f5180003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51977fe640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51800039b0 0x7f5180003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51977fe640 1 -- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51800043a0 msgr2=0x7f5180004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51977fe640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51800043a0 0x7f5180004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.437+0000 7f51977fe640 1 -- 172.21.15.6:0/2242961142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5180003520 con 0x7f5180004ef0 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.438+0000 7f51977fe640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5180004ef0 0x7f518000b9c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f519000a8a0 tx=0x7f5180005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d586f923ca8979a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.438+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519000a4c0 con 0x7f5180004ef0 2024-03-18T14:44:21.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.438+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5190034650 con 0x7f5180004ef0 2024-03-18T14:44:21.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.438+0000 7f5195ffb640 1 -- 172.21.15.6:0/2242961142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190034a80 con 0x7f5180004ef0 2024-03-18T14:44:21.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.439+0000 7f519d861640 1 -- 172.21.15.6:0/2242961142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5180004ef0 msgr2=0x7f518000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.439+0000 7f519d861640 1 --2- 172.21.15.6:0/2242961142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5180004ef0 0x7f518000b9c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f519000a8a0 tx=0x7f5180005830 comp rx=0 tx=0).stop 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.439+0000 7f51977fe640 1 -- 172.21.15.6:0/2242961142 reap_dead start 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.439+0000 7f519d861640 1 -- 172.21.15.6:0/2242961142 shutdown_connections 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.439+0000 7f519d861640 1 -- 172.21.15.6:0/2242961142 >> 172.21.15.6:0/2242961142 conn(0x7f51980b1500 msgr2=0x7f51980b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.439+0000 7f519d861640 1 -- 172.21.15.6:0/2242961142 shutdown_connections 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.439+0000 7f519d861640 1 -- 172.21.15.6:0/2242961142 wait complete. 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.440+0000 7f519d861640 1 Processor -- start 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.440+0000 7f519d861640 1 -- start start 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f519d861640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5180004ef0 0x7f51981594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f519d861640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51980ae210 0x7f51980ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f519d861640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51980abbd0 0x7f51980ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f519d861640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f51980cefd0 con 0x7f51980abbd0 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f519d861640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f519806dad0 con 0x7f5180004ef0 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f519d861640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f519806f000 con 0x7f51980ae210 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51980abbd0 0x7f51980ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51980abbd0 0x7f51980ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41534/0 (socket says 172.21.15.6:41534) 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 -- 172.21.15.6:0/2373041818 learned_addr learned my addr 172.21.15.6:0/2373041818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:21.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51967fc640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51980ae210 0x7f51980ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f5196ffd640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5180004ef0 0x7f51981594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 -- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51980ae210 msgr2=0x7f51980ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51980ae210 0x7f51980ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 -- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5180004ef0 msgr2=0x7f51981594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5180004ef0 0x7f51981594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 -- 172.21.15.6:0/2373041818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51980ac890 con 0x7f51980abbd0 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51967fc640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f51980ae210 0x7f51980ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:44:21.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f5196ffd640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5180004ef0 0x7f51981594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.441+0000 7f51977fe640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51980abbd0 0x7f51980ac050 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5190001cf0 tx=0x7f5190001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.442+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190002010 con 0x7f51980abbd0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.442+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5190044070 con 0x7f51980abbd0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.442+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51980acb70 con 0x7f51980abbd0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.442+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190034c00 con 0x7f51980abbd0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.442+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51980ad130 con 0x7f51980abbd0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.442+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5198073330 con 0x7f51980abbd0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.443+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5190049460 con 0x7f51980abbd0 2024-03-18T14:44:21.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.443+0000 7f517ffff640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f518406bbd0 0x7f518406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f51900acd40 con 0x7f51980abbd0 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f517ffff640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7f5184072070 0x7f51840744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 --> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5190031ec0 con 0x7f5184072070 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f51900af440 con 0x7f51980abbd0 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f51967fc640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7f5184072070 0x7f51840744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f5196ffd640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f518406bbd0 0x7f518406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f51967fc640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7f5184072070 0x7f51840744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.444+0000 7f5196ffd640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f518406bbd0 0x7f518406e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f518c009fb0 tx=0x7f518c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.445+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== osd.1 v2:172.21.15.6:6809/2441154404 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5190031ec0 con 0x7f5184072070 2024-03-18T14:44:21.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.458+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 --> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f519804f620 con 0x7f5184072070 2024-03-18T14:44:21.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.458+0000 7f517ffff640 1 -- 172.21.15.6:0/2373041818 <== osd.1 v2:172.21.15.6:6809/2441154404 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f519804f620 con 0x7f5184072070 2024-03-18T14:44:21.460 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:21.460 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:21.460 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:21.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.458+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7f5184072070 msgr2=0x7f51840744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.458+0000 7f519d861640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:6809/2441154404,v1:172.21.15.6:6810/2441154404] conn(0x7f5184072070 0x7f51840744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f518406bbd0 msgr2=0x7f518406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f518406bbd0 0x7f518406e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f518c009fb0 tx=0x7f518c009190 comp rx=0 tx=0).stop 2024-03-18T14:44:21.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51980abbd0 msgr2=0x7f51980ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 --2- 172.21.15.6:0/2373041818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51980abbd0 0x7f51980ac050 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5190001cf0 tx=0x7f5190001dc0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f51977fe640 1 -- 172.21.15.6:0/2373041818 reap_dead start 2024-03-18T14:44:21.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 shutdown_connections 2024-03-18T14:44:21.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 >> 172.21.15.6:0/2373041818 conn(0x7f51980b1500 msgr2=0x7f519815a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 shutdown_connections 2024-03-18T14:44:21.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.459+0000 7f519d861640 1 -- 172.21.15.6:0/2373041818 wait complete. 2024-03-18T14:44:21.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-18T14:44:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.573+0000 7f42c7cb2640 1 Processor -- start 2024-03-18T14:44:21.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.573+0000 7f42c7cb2640 1 -- start start 2024-03-18T14:44:21.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c7cb2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f42c00b49b0 0x7f42c00cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c7cb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c006b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c7cb2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42c00cfe50 con 0x7f42c00b4d90 2024-03-18T14:44:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c7cb2640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f42c015c0c0 con 0x7f42c00b49b0 2024-03-18T14:44:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c7cb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f42c01555d0 con 0x7f42c006b1c0 2024-03-18T14:44:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c5a4e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f42c00b49b0 0x7f42c00cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c5a4e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f42c00b49b0 0x7f42c00cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41546/0 (socket says 172.21.15.6:41546) 2024-03-18T14:44:21.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c5a4e640 1 -- 172.21.15.6:0/2019142039 learned_addr learned my addr 172.21.15.6:0/2019142039 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c624f640 1 --2- 172.21.15.6:0/2019142039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c006b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c5a4e640 1 -- 172.21.15.6:0/2019142039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 msgr2=0x7f42c006b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c5a4e640 1 --2- 172.21.15.6:0/2019142039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c006b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c5a4e640 1 -- 172.21.15.6:0/2019142039 >> v1:172.21.15.92:6789/0 conn(0x7f42c00b4d90 legacy=0x7f42c015b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.574+0000 7f42c5a4e640 1 -- 172.21.15.6:0/2019142039 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c00ce870 con 0x7f42c00b49b0 2024-03-18T14:44:21.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.575+0000 7f42c5a4e640 1 --2- 172.21.15.6:0/2019142039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c00cf910 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00bb10 tx=0x7f42bc02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8f68e4ae238bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.575+0000 7f42c4a4c640 1 -- 172.21.15.6:0/2019142039 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc03b070 con 0x7f42c00b49b0 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.575+0000 7f42c4a4c640 1 -- 172.21.15.6:0/2019142039 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42bc036440 con 0x7f42c00b49b0 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.575+0000 7f42c4a4c640 1 -- 172.21.15.6:0/2019142039 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc036810 con 0x7f42c00b49b0 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 -- 172.21.15.6:0/2019142039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 msgr2=0x7f42c00cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/2019142039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c00cf910 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00bb10 tx=0x7f42bc02e150 comp rx=0 tx=0).stop 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 -- 172.21.15.6:0/2019142039 shutdown_connections 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/2019142039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c006b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/2019142039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c00cf910 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 -- 172.21.15.6:0/2019142039 >> 172.21.15.6:0/2019142039 conn(0x7f42c006e330 msgr2=0x7f42c006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 -- 172.21.15.6:0/2019142039 shutdown_connections 2024-03-18T14:44:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.576+0000 7f42c7cb2640 1 -- 172.21.15.6:0/2019142039 wait complete. 2024-03-18T14:44:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.577+0000 7f42c7cb2640 1 Processor -- start 2024-03-18T14:44:21.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.577+0000 7f42c7cb2640 1 -- start start 2024-03-18T14:44:21.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c7cb2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c01f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c7cb2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c0140c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c7cb2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c00b4d90 0x7f42c0159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c7cb2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00cf110 con 0x7f42c00b49b0 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c7cb2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00cee30 con 0x7f42c00b4d90 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c7cb2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f42c00cf3f0 con 0x7f42c006b1c0 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c0140c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c0140c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41554/0 (socket says 172.21.15.6:41554) 2024-03-18T14:44:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 -- 172.21.15.6:0/768062216 learned_addr learned my addr 172.21.15.6:0/768062216 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c5a4e640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c01f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c624f640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c00b4d90 0x7f42c0159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 -- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 msgr2=0x7f42c01f7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c01f7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 -- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c00b4d90 msgr2=0x7f42c0159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c00b4d90 0x7f42c0159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 -- 172.21.15.6:0/768062216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c01f83c0 con 0x7f42c00b49b0 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c5a4e640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c01f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:21.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c624f640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c00b4d90 0x7f42c0159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.578+0000 7f42c524d640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c0140c50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f42b000c3c0 tx=0x7f42b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.579+0000 7f42b67fc640 1 -- 172.21.15.6:0/768062216 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b00052d0 con 0x7f42c00b49b0 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.579+0000 7f42b67fc640 1 -- 172.21.15.6:0/768062216 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b0005490 con 0x7f42c00b49b0 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.579+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c0143880 con 0x7f42c00b49b0 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.579+0000 7f42b67fc640 1 -- 172.21.15.6:0/768062216 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0007260 con 0x7f42c00b49b0 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.579+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c0143e40 con 0x7f42c00b49b0 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.580+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c004f5b0 con 0x7f42c00b49b0 2024-03-18T14:44:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.580+0000 7f42b67fc640 1 -- 172.21.15.6:0/768062216 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f42b0007400 con 0x7f42c00b49b0 2024-03-18T14:44:21.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.581+0000 7f42b67fc640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f42a406bbd0 0x7f42a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.581+0000 7f42b67fc640 1 -- 172.21.15.6:0/768062216 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f42b0089d80 con 0x7f42c00b49b0 2024-03-18T14:44:21.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.581+0000 7f42c5a4e640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f42a406bbd0 0x7f42a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.582+0000 7f42c5a4e640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f42a406bbd0 0x7f42a406e090 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f42bc038000 tx=0x7f42bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.585+0000 7f42b67fc640 1 -- 172.21.15.6:0/768062216 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f42c01f83c0 con 0x7f42c00b49b0 2024-03-18T14:44:21.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.769+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f42c006efa0 con 0x7f42a406bbd0 2024-03-18T14:44:21.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.769+0000 7f42b67fc640 1 -- 172.21.15.6:0/768062216 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f42c006efa0 con 0x7f42a406bbd0 2024-03-18T14:44:21.771 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-03-18T14:44:21.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f42a406bbd0 msgr2=0x7f42a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f42a406bbd0 0x7f42a406e090 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f42bc038000 tx=0x7f42bc008040 comp rx=0 tx=0).stop 2024-03-18T14:44:21.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 msgr2=0x7f42c0140c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c0140c50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f42b000c3c0 tx=0x7f42b000c880 comp rx=0 tx=0).stop 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 shutdown_connections 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f42a406bbd0 0x7f42a406e090 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c00b4d90 0x7f42c0159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c00b49b0 0x7f42c0140c50 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 --2- 172.21.15.6:0/768062216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42c006b1c0 0x7f42c01f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 >> 172.21.15.6:0/768062216 conn(0x7f42c006e330 msgr2=0x7f42c00afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 shutdown_connections 2024-03-18T14:44:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.772+0000 7f42c7cb2640 1 -- 172.21.15.6:0/768062216 wait complete. 2024-03-18T14:44:21.784 INFO:tasks.ceph:Scrubbing osd.2 2024-03-18T14:44:21.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.889+0000 7f6347a0b640 1 Processor -- start 2024-03-18T14:44:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.889+0000 7f6347a0b640 1 -- start start 2024-03-18T14:44:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.889+0000 7f6347a0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400d1de0 0x7f63400d3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.889+0000 7f6347a0b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f634006c730 0x7f634006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.889+0000 7f6347a0b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63400d22c0 con 0x7f634015e0b0 2024-03-18T14:44:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6347a0b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f63400d25a0 con 0x7f634006c730 2024-03-18T14:44:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6347a0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f63400d2880 con 0x7f63400d1de0 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6345fa8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f634006c730 0x7f634006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6345fa8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f634006c730 0x7f634006cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41564/0 (socket says 172.21.15.6:41564) 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6345fa8640 1 -- 172.21.15.6:0/1178064784 learned_addr learned my addr 172.21.15.6:0/1178064784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6344fa6640 1 --2- 172.21.15.6:0/1178064784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400d1de0 0x7f63400d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6345fa8640 1 -- 172.21.15.6:0/1178064784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400d1de0 msgr2=0x7f63400d3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6345fa8640 1 --2- 172.21.15.6:0/1178064784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400d1de0 0x7f63400d3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6345fa8640 1 -- 172.21.15.6:0/1178064784 >> v1:172.21.15.92:6789/0 conn(0x7f634015e0b0 legacy=0x7f63400b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.890+0000 7f6345fa8640 1 -- 172.21.15.6:0/1178064784 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63400d2b60 con 0x7f634006c730 2024-03-18T14:44:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6345fa8640 1 --2- 172.21.15.6:0/1178064784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f634006c730 0x7f634006cbe0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6338009e50 tx=0x7f633802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bfa4287c688f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6337fff640 1 -- 172.21.15.6:0/1178064784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633802cb90 con 0x7f634006c730 2024-03-18T14:44:21.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6337fff640 1 -- 172.21.15.6:0/1178064784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633802cd30 con 0x7f634006c730 2024-03-18T14:44:21.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6337fff640 1 -- 172.21.15.6:0/1178064784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338034610 con 0x7f634006c730 2024-03-18T14:44:21.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6347a0b640 1 -- 172.21.15.6:0/1178064784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f634006c730 msgr2=0x7f634006cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6347a0b640 1 --2- 172.21.15.6:0/1178064784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f634006c730 0x7f634006cbe0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6338009e50 tx=0x7f633802bff0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6347a0b640 1 -- 172.21.15.6:0/1178064784 shutdown_connections 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6347a0b640 1 --2- 172.21.15.6:0/1178064784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f634006c730 0x7f634006cbe0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6347a0b640 1 --2- 172.21.15.6:0/1178064784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63400d1de0 0x7f63400d3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.891+0000 7f6347a0b640 1 -- 172.21.15.6:0/1178064784 >> 172.21.15.6:0/1178064784 conn(0x7f6340072ce0 msgr2=0x7f63400730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.892+0000 7f6347a0b640 1 -- 172.21.15.6:0/1178064784 shutdown_connections 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.892+0000 7f6347a0b640 1 -- 172.21.15.6:0/1178064784 wait complete. 2024-03-18T14:44:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 Processor -- start 2024-03-18T14:44:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 -- start start 2024-03-18T14:44:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634006c730 0x7f6340160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400d1de0 0x7f63401589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634015e0b0 0x7f634015a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f63400d38a0 con 0x7f63400d1de0 2024-03-18T14:44:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f63400b6d00 con 0x7f634006c730 2024-03-18T14:44:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.893+0000 7f6347a0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f634015b260 con 0x7f634015e0b0 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400d1de0 0x7f63401589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400d1de0 0x7f63401589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41580/0 (socket says 172.21.15.6:41580) 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 -- 172.21.15.6:0/793030678 learned_addr learned my addr 172.21.15.6:0/793030678 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f63457a7640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634006c730 0x7f6340160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6345fa8640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634015e0b0 0x7f634015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 -- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634015e0b0 msgr2=0x7f634015a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f634015e0b0 0x7f634015a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 -- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634006c730 msgr2=0x7f6340160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634006c730 0x7f6340160270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 -- 172.21.15.6:0/793030678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634015b540 con 0x7f63400d1de0 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6344fa6640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400d1de0 0x7f63401589a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f633000d800 tx=0x7f633000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633000b6c0 con 0x7f63400d1de0 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6330019070 con 0x7f63400d1de0 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633000ba80 con 0x7f63400d1de0 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.894+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634015b820 con 0x7f63400d1de0 2024-03-18T14:44:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.895+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6340158ee0 con 0x7f63400d1de0 2024-03-18T14:44:21.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.895+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f634008afe0 con 0x7f63400d1de0 2024-03-18T14:44:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.896+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f633000bc20 con 0x7f63400d1de0 2024-03-18T14:44:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.896+0000 7f6335ffb640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632406bbd0 0x7f632406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f633008adf0 con 0x7f63400d1de0 2024-03-18T14:44:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f6335ffb640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6324072070 0x7f63240744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 --> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6330001d30 con 0x7f6324072070 2024-03-18T14:44:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f633008d4f0 con 0x7f63400d1de0 2024-03-18T14:44:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f63457a7640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632406bbd0 0x7f632406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f6345fa8640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6324072070 0x7f63240744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f6345fa8640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6324072070 0x7f63240744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.897+0000 7f63457a7640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632406bbd0 0x7f632406e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f633c007fd0 tx=0x7f633c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.898+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== osd.2 v2:172.21.15.6:6800/3396973445 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6330001d30 con 0x7f6324072070 2024-03-18T14:44:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.911+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 --> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f634004f620 con 0x7f6324072070 2024-03-18T14:44:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.911+0000 7f6335ffb640 1 -- 172.21.15.6:0/793030678 <== osd.2 v2:172.21.15.6:6800/3396973445 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f634004f620 con 0x7f6324072070 2024-03-18T14:44:21.913 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:21.913 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:21.913 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:21.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.911+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6324072070 msgr2=0x7f63240744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:6800/3396973445,v1:172.21.15.6:6801/3396973445] conn(0x7f6324072070 0x7f63240744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632406bbd0 msgr2=0x7f632406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f632406bbd0 0x7f632406e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f633c007fd0 tx=0x7f633c009040 comp rx=0 tx=0).stop 2024-03-18T14:44:21.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400d1de0 msgr2=0x7f63401589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:21.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 --2- 172.21.15.6:0/793030678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f63400d1de0 0x7f63401589a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f633000d800 tx=0x7f633000dcc0 comp rx=0 tx=0).stop 2024-03-18T14:44:21.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6345fa8640 1 -- 172.21.15.6:0/793030678 reap_dead start 2024-03-18T14:44:21.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 shutdown_connections 2024-03-18T14:44:21.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 >> 172.21.15.6:0/793030678 conn(0x7f6340072ce0 msgr2=0x7f63400b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:21.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 shutdown_connections 2024-03-18T14:44:21.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:21.912+0000 7f6347a0b640 1 -- 172.21.15.6:0/793030678 wait complete. 2024-03-18T14:44:21.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-18T14:44:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.026+0000 7fae4c2ff640 1 Processor -- start 2024-03-18T14:44:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.026+0000 7fae4c2ff640 1 -- start start 2024-03-18T14:44:22.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4c2ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4415e230 0x7fae4406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4c2ff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fae440d04d0 0x7fae4415e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4c2ff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae44161de0 con 0x7fae4406b700 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4c2ff640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fae44161b90 con 0x7fae440d04d0 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4c2ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fae441620c0 con 0x7fae4415e230 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4a89c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fae440d04d0 0x7fae4415e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4989a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fae4406b700 0x7fae440cfdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55504/0 (socket says 172.21.15.6:55504) 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4989a640 1 -- 172.21.15.6:0/188968969 learned_addr learned my addr 172.21.15.6:0/188968969 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4a09b640 1 --2- 172.21.15.6:0/188968969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4415e230 0x7fae4406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4a89c640 1 -- 172.21.15.6:0/188968969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4415e230 msgr2=0x7fae4406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:22.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4a89c640 1 --2- 172.21.15.6:0/188968969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4415e230 0x7fae4406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4a89c640 1 -- 172.21.15.6:0/188968969 >> v1:172.21.15.6:6789/0 conn(0x7fae4406b700 legacy=0x7fae440cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae4a89c640 1 -- 172.21.15.6:0/188968969 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae441623a0 con 0x7fae440d04d0 2024-03-18T14:44:22.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.027+0000 7fae49099640 1 -- 172.21.15.6:0/188968969 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 183633301 0 0) 0x7fae44161de0 con 0x7fae4406b700 2024-03-18T14:44:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.028+0000 7fae4a89c640 1 --2- 172.21.15.6:0/188968969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440d04d0 0x7fae4415e620 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009d90 tx=0x7fae3c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760d83740a2e1cde server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.028+0000 7fae49099640 1 -- 172.21.15.6:0/188968969 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c039070 con 0x7fae440d04d0 2024-03-18T14:44:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.028+0000 7fae49099640 1 -- 172.21.15.6:0/188968969 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae3c02cbc0 con 0x7fae440d04d0 2024-03-18T14:44:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.028+0000 7fae49099640 1 -- 172.21.15.6:0/188968969 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c034440 con 0x7fae440d04d0 2024-03-18T14:44:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 -- 172.21.15.6:0/188968969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440d04d0 msgr2=0x7fae4415e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/188968969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440d04d0 0x7fae4415e620 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009d90 tx=0x7fae3c02be10 comp rx=0 tx=0).stop 2024-03-18T14:44:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 -- 172.21.15.6:0/188968969 shutdown_connections 2024-03-18T14:44:22.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/188968969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440d04d0 0x7fae4415e620 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/188968969 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4415e230 0x7fae4406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 -- 172.21.15.6:0/188968969 >> 172.21.15.6:0/188968969 conn(0x7fae44099a50 msgr2=0x7fae440d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 -- 172.21.15.6:0/188968969 shutdown_connections 2024-03-18T14:44:22.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.029+0000 7fae4c2ff640 1 -- 172.21.15.6:0/188968969 wait complete. 2024-03-18T14:44:22.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.030+0000 7fae4c2ff640 1 Processor -- start 2024-03-18T14:44:22.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.030+0000 7fae4c2ff640 1 -- start start 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4c2ff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4406b700 0x7fae441f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4c2ff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae440d04d0 0x7fae4414af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4c2ff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440c9cd0 0x7fae440ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4c2ff640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae440cb080 con 0x7fae440c9cd0 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4c2ff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fae440d15c0 con 0x7fae440d04d0 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4c2ff640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fae440cb200 con 0x7fae4406b700 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440c9cd0 0x7fae440ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440c9cd0 0x7fae440ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41608/0 (socket says 172.21.15.6:41608) 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 -- 172.21.15.6:0/274091373 learned_addr learned my addr 172.21.15.6:0/274091373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a09b640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4406b700 0x7fae441f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4989a640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae440d04d0 0x7fae4414af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 -- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4406b700 msgr2=0x7fae441f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4406b700 0x7fae441f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 -- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae440d04d0 msgr2=0x7fae4414af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae440d04d0 0x7fae4414af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 -- 172.21.15.6:0/274091373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae440c51e0 con 0x7fae440c9cd0 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a09b640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4406b700 0x7fae441f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae4a89c640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440c9cd0 0x7fae440ca160 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009d90 tx=0x7fae3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.031+0000 7fae3affd640 1 -- 172.21.15.6:0/274091373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c039050 con 0x7fae440c9cd0 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.032+0000 7fae3affd640 1 -- 172.21.15.6:0/274091373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae3c045070 con 0x7fae440c9cd0 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.032+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae440c54c0 con 0x7fae440c9cd0 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.032+0000 7fae3affd640 1 -- 172.21.15.6:0/274091373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c009450 con 0x7fae440c9cd0 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.032+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae440c5a80 con 0x7fae440c9cd0 2024-03-18T14:44:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.033+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae440681a0 con 0x7fae440c9cd0 2024-03-18T14:44:22.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.033+0000 7fae3affd640 1 -- 172.21.15.6:0/274091373 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fae3c03e850 con 0x7fae440c9cd0 2024-03-18T14:44:22.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.033+0000 7fae3affd640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae2806bc00 0x7fae2806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.034+0000 7fae3affd640 1 -- 172.21.15.6:0/274091373 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7fae3c0ad580 con 0x7fae440c9cd0 2024-03-18T14:44:22.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.034+0000 7fae4a09b640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae2806bc00 0x7fae2806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.034+0000 7fae4a09b640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae2806bc00 0x7fae2806e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fae440b57b0 tx=0x7fae40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.038+0000 7fae3affd640 1 -- 172.21.15.6:0/274091373 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fae440681a0 con 0x7fae440c9cd0 2024-03-18T14:44:22.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.222+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fae4404f5b0 con 0x7fae2806bc00 2024-03-18T14:44:22.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.222+0000 7fae3affd640 1 -- 172.21.15.6:0/274091373 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fae4404f5b0 con 0x7fae2806bc00 2024-03-18T14:44:22.224 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-03-18T14:44:22.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae2806bc00 msgr2=0x7fae2806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae2806bc00 0x7fae2806e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fae440b57b0 tx=0x7fae40008040 comp rx=0 tx=0).stop 2024-03-18T14:44:22.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440c9cd0 msgr2=0x7fae440ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440c9cd0 0x7fae440ca160 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009d90 tx=0x7fae3c004a40 comp rx=0 tx=0).stop 2024-03-18T14:44:22.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 shutdown_connections 2024-03-18T14:44:22.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fae2806bc00 0x7fae2806e0c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae440c9cd0 0x7fae440ca160 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae440d04d0 0x7fae4414af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 --2- 172.21.15.6:0/274091373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae4406b700 0x7fae441f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 >> 172.21.15.6:0/274091373 conn(0x7fae44099a50 msgr2=0x7fae440cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 shutdown_connections 2024-03-18T14:44:22.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.224+0000 7fae4c2ff640 1 -- 172.21.15.6:0/274091373 wait complete. 2024-03-18T14:44:22.236 INFO:tasks.ceph:Scrubbing osd.3 2024-03-18T14:44:22.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.341+0000 7fc97c412640 1 Processor -- start 2024-03-18T14:44:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.341+0000 7fc97c412640 1 -- start start 2024-03-18T14:44:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc97c412640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc97415e000 0x7fc9740b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc97c412640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc97c412640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9740b1ca0 con 0x7fc9740ce3b0 2024-03-18T14:44:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc97c412640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fc9740b28b0 con 0x7fc97415e000 2024-03-18T14:44:22.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc97c412640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc9740d23d0 con 0x7fc9740d1a50 2024-03-18T14:44:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc9799ad640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc97a1ae640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc97415e000 0x7fc9740b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc9799ad640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34774/0 (socket says 172.21.15.6:34774) 2024-03-18T14:44:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc9799ad640 1 -- 172.21.15.6:0/186462701 learned_addr learned my addr 172.21.15.6:0/186462701 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc9799ad640 1 -- 172.21.15.6:0/186462701 >> v2:172.21.15.92:3300/0 conn(0x7fc97415e000 msgr2=0x7fc9740b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc9799ad640 1 --2- 172.21.15.6:0/186462701 >> v2:172.21.15.92:3300/0 conn(0x7fc97415e000 0x7fc9740b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc9799ad640 1 -- 172.21.15.6:0/186462701 >> v1:172.21.15.92:6789/0 conn(0x7fc9740ce3b0 legacy=0x7fc9740ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc9799ad640 1 -- 172.21.15.6:0/186462701 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9740b1f80 con 0x7fc9740d1a50 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.342+0000 7fc97a1ae640 1 --2- 172.21.15.6:0/186462701 >> v2:172.21.15.92:3300/0 conn(0x7fc97415e000 0x7fc9740b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.343+0000 7fc9799ad640 1 --2- 172.21.15.6:0/186462701 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740d1e90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc964009870 tx=0x7fc96402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a72bd99fc30c3f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.343+0000 7fc9791ac640 1 -- 172.21.15.6:0/186462701 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96402f840 con 0x7fc9740d1a50 2024-03-18T14:44:22.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.343+0000 7fc9791ac640 1 -- 172.21.15.6:0/186462701 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc96402f9e0 con 0x7fc9740d1a50 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.343+0000 7fc9791ac640 1 -- 172.21.15.6:0/186462701 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96402fdb0 con 0x7fc9740d1a50 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 -- 172.21.15.6:0/186462701 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 msgr2=0x7fc9740d1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 --2- 172.21.15.6:0/186462701 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740d1e90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc964009870 tx=0x7fc96402ba20 comp rx=0 tx=0).stop 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 -- 172.21.15.6:0/186462701 shutdown_connections 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 --2- 172.21.15.6:0/186462701 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740d1e90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 --2- 172.21.15.6:0/186462701 >> v2:172.21.15.92:3300/0 conn(0x7fc97415e000 0x7fc9740b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 -- 172.21.15.6:0/186462701 >> 172.21.15.6:0/186462701 conn(0x7fc97409ae00 msgr2=0x7fc97408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 -- 172.21.15.6:0/186462701 shutdown_connections 2024-03-18T14:44:22.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.344+0000 7fc97c412640 1 -- 172.21.15.6:0/186462701 wait complete. 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.345+0000 7fc97c412640 1 Processor -- start 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.345+0000 7fc97c412640 1 -- start start 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97c412640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9740ce3b0 0x7fc974160f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97c412640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97c412640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc97415e000 0x7fc9740b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97c412640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc9740b2540 con 0x7fc9740ce3b0 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97c412640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc97415e3e0 con 0x7fc97415e000 2024-03-18T14:44:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97c412640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fc9740b1210 con 0x7fc9740d1a50 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9740ce3b0 0x7fc974160f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9740ce3b0 0x7fc974160f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41620/0 (socket says 172.21.15.6:41620) 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 -- 172.21.15.6:0/3829951056 learned_addr learned my addr 172.21.15.6:0/3829951056 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc9799ad640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a9af640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc97415e000 0x7fc9740b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 -- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 msgr2=0x7fc9740afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc9740d1a50 0x7fc9740afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 -- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc97415e000 msgr2=0x7fc9740b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc97415e000 0x7fc9740b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 -- 172.21.15.6:0/3829951056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9741f4b50 con 0x7fc9740ce3b0 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.346+0000 7fc97a1ae640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9740ce3b0 0x7fc974160f00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e990 tx=0x7fc97000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.347+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970017070 con 0x7fc9740ce3b0 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.347+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc97000c9e0 con 0x7fc9740ce3b0 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.347+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9740b47c0 con 0x7fc9740ce3b0 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.347+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97000cdb0 con 0x7fc9740ce3b0 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.347+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9740b4d80 con 0x7fc9740ce3b0 2024-03-18T14:44:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.347+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc97409fa70 con 0x7fc9740ce3b0 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.348+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc97001a070 con 0x7fc9740ce3b0 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.348+0000 7fc96affd640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc95806bbd0 0x7fc95806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7fc97008a760 con 0x7fc9740ce3b0 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc9799ad640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc95806bbd0 0x7fc95806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc96affd640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7fc958072070 0x7fc9580744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 --> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc970022e50 con 0x7fc958072070 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fc970091050 con 0x7fc9740ce3b0 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc97a9af640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7fc958072070 0x7fc9580744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc97a9af640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7fc958072070 0x7fc9580744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.349+0000 7fc9799ad640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc95806bbd0 0x7fc95806e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc964036000 tx=0x7fc964008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.350+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== osd.3 v2:172.21.15.6:6808/536857262 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc970022e50 con 0x7fc958072070 2024-03-18T14:44:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 --> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc97404f620 con 0x7fc958072070 2024-03-18T14:44:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc96affd640 1 -- 172.21.15.6:0/3829951056 <== osd.3 v2:172.21.15.6:6808/536857262 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc97404f620 con 0x7fc958072070 2024-03-18T14:44:22.364 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:22.365 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:22.365 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7fc958072070 msgr2=0x7fc9580744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc97c412640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:6808/536857262,v1:172.21.15.6:6811/536857262] conn(0x7fc958072070 0x7fc9580744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc95806bbd0 msgr2=0x7fc95806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc97c412640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fc95806bbd0 0x7fc95806e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc964036000 tx=0x7fc964008030 comp rx=0 tx=0).stop 2024-03-18T14:44:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9740ce3b0 msgr2=0x7fc974160f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.363+0000 7fc97c412640 1 --2- 172.21.15.6:0/3829951056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc9740ce3b0 0x7fc974160f00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e990 tx=0x7fc97000ee50 comp rx=0 tx=0).stop 2024-03-18T14:44:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.364+0000 7fc97a9af640 1 -- 172.21.15.6:0/3829951056 reap_dead start 2024-03-18T14:44:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.364+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 shutdown_connections 2024-03-18T14:44:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.364+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 >> 172.21.15.6:0/3829951056 conn(0x7fc97409ae00 msgr2=0x7fc9740cd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.364+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 shutdown_connections 2024-03-18T14:44:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.364+0000 7fc97c412640 1 -- 172.21.15.6:0/3829951056 wait complete. 2024-03-18T14:44:22.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-18T14:44:22.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.476+0000 7f7268c0b640 1 Processor -- start 2024-03-18T14:44:22.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.476+0000 7f7268c0b640 1 -- start start 2024-03-18T14:44:22.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7268c0b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f726415a440 0x7f72640721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7268c0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72640b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7268c0b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726406da60 con 0x7f72640b40d0 2024-03-18T14:44:22.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7268c0b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f7264072b70 con 0x7f726415a440 2024-03-18T14:44:22.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7268c0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f726406dd40 con 0x7f7264072730 2024-03-18T14:44:22.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f726259c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f726415a440 0x7f72640721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7261d9b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72640b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f726259c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f726415a440 0x7f72640721f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50272/0 (socket says 172.21.15.6:50272) 2024-03-18T14:44:22.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f726259c640 1 -- 172.21.15.6:0/2087597539 learned_addr learned my addr 172.21.15.6:0/2087597539 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7261d9b640 1 -- 172.21.15.6:0/2087597539 >> v2:172.21.15.92:3300/0 conn(0x7f726415a440 msgr2=0x7f72640721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:22.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.477+0000 7f7261d9b640 1 --2- 172.21.15.6:0/2087597539 >> v2:172.21.15.92:3300/0 conn(0x7f726415a440 0x7f72640721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.478+0000 7f7261d9b640 1 -- 172.21.15.6:0/2087597539 >> v1:172.21.15.92:6789/0 conn(0x7f72640b40d0 legacy=0x7f72640b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.478+0000 7f7261d9b640 1 -- 172.21.15.6:0/2087597539 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726406e020 con 0x7f7264072730 2024-03-18T14:44:22.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.478+0000 7f726259c640 1 --2- 172.21.15.6:0/2087597539 >> v2:172.21.15.92:3300/0 conn(0x7f726415a440 0x7f72640721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T14:44:22.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.478+0000 7f7261d9b640 1 --2- 172.21.15.6:0/2087597539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72640b3b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f72580097a0 tx=0x7f725802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8549cd5b4b9f431 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.478+0000 7f726159a640 1 -- 172.21.15.6:0/2087597539 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258035020 con 0x7f7264072730 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f726159a640 1 -- 172.21.15.6:0/2087597539 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7258031830 con 0x7f7264072730 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f726159a640 1 -- 172.21.15.6:0/2087597539 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258031c00 con 0x7f7264072730 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 -- 172.21.15.6:0/2087597539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 msgr2=0x7f72640b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 --2- 172.21.15.6:0/2087597539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72640b3b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f72580097a0 tx=0x7f725802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 -- 172.21.15.6:0/2087597539 shutdown_connections 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 --2- 172.21.15.6:0/2087597539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72640b3b90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 --2- 172.21.15.6:0/2087597539 >> v2:172.21.15.92:3300/0 conn(0x7f726415a440 0x7f72640721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 -- 172.21.15.6:0/2087597539 >> 172.21.15.6:0/2087597539 conn(0x7f72640b1570 msgr2=0x7f72640b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 -- 172.21.15.6:0/2087597539 shutdown_connections 2024-03-18T14:44:22.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.479+0000 7f7268c0b640 1 -- 172.21.15.6:0/2087597539 wait complete. 2024-03-18T14:44:22.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.480+0000 7f7268c0b640 1 Processor -- start 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7268c0b640 1 -- start start 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7268c0b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72641f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7268c0b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72640b40d0 0x7f7264071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7268c0b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f726415a440 0x7f7264071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7268c0b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f72640b4520 con 0x7f726415a440 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7268c0b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f72640ce3d0 con 0x7f72640b40d0 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7268c0b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f72640b4810 con 0x7f7264072730 2024-03-18T14:44:22.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7262d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f726415a440 0x7f7264071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7262d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f726415a440 0x7f7264071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41628/0 (socket says 172.21.15.6:41628) 2024-03-18T14:44:22.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7262d9d640 1 -- 172.21.15.6:0/3346730431 learned_addr learned my addr 172.21.15.6:0/3346730431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f726259c640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72641f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.481+0000 7f7261d9b640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72640b40d0 0x7f7264071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7262d9d640 1 -- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 msgr2=0x7f72641f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7262d9d640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72641f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7262d9d640 1 -- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72640b40d0 msgr2=0x7f7264071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7262d9d640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72640b40d0 0x7f7264071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7262d9d640 1 -- 172.21.15.6:0/3346730431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72640ab0e0 con 0x7f726415a440 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7262d9d640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f726415a440 0x7f7264071ea0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f725c00c7d0 tx=0x7f725c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7252ffd640 1 -- 172.21.15.6:0/3346730431 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c003b70 con 0x7f726415a440 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7252ffd640 1 -- 172.21.15.6:0/3346730431 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725c003d30 con 0x7f726415a440 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72640ab3c0 con 0x7f726415a440 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7252ffd640 1 -- 172.21.15.6:0/3346730431 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c00aa90 con 0x7f726415a440 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.482+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72640ab980 con 0x7f726415a440 2024-03-18T14:44:22.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.483+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72640b4ad0 con 0x7f726415a440 2024-03-18T14:44:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.483+0000 7f7252ffd640 1 -- 172.21.15.6:0/3346730431 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f725c00ac30 con 0x7f726415a440 2024-03-18T14:44:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.484+0000 7f7252ffd640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f724406bbd0 0x7f724406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.484+0000 7f7252ffd640 1 -- 172.21.15.6:0/3346730431 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f725c08a640 con 0x7f726415a440 2024-03-18T14:44:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.484+0000 7f726259c640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f724406bbd0 0x7f724406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.484+0000 7f726259c640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f724406bbd0 0x7f724406e090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7254006ca0 tx=0x7f7254009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.488+0000 7f7252ffd640 1 -- 172.21.15.6:0/3346730431 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f72640b4ad0 con 0x7f726415a440 2024-03-18T14:44:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.671+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f72640cc0b0 con 0x7f724406bbd0 2024-03-18T14:44:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.672+0000 7f7252ffd640 1 -- 172.21.15.6:0/3346730431 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f72640cc0b0 con 0x7f724406bbd0 2024-03-18T14:44:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2024-03-18T14:44:22.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f724406bbd0 msgr2=0x7f724406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f724406bbd0 0x7f724406e090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7254006ca0 tx=0x7f7254009240 comp rx=0 tx=0).stop 2024-03-18T14:44:22.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f726415a440 msgr2=0x7f7264071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f726415a440 0x7f7264071ea0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f725c00c7d0 tx=0x7f725c00cc90 comp rx=0 tx=0).stop 2024-03-18T14:44:22.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 shutdown_connections 2024-03-18T14:44:22.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f724406bbd0 0x7f724406e090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f726415a440 0x7f7264071ea0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72640b40d0 0x7f7264071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 --2- 172.21.15.6:0/3346730431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7264072730 0x7f72641f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 >> 172.21.15.6:0/3346730431 conn(0x7f72640b1570 msgr2=0x7f72641583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 shutdown_connections 2024-03-18T14:44:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.674+0000 7f7268c0b640 1 -- 172.21.15.6:0/3346730431 wait complete. 2024-03-18T14:44:22.686 INFO:tasks.ceph:Scrubbing osd.4 2024-03-18T14:44:22.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.790+0000 7f066baf7640 1 Processor -- start 2024-03-18T14:44:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.790+0000 7f066baf7640 1 -- start start 2024-03-18T14:44:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066baf7640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06640d1de0 0x7f06640d3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066baf7640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066406c730 0x7f066406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066baf7640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06640d22c0 con 0x7f066415e0b0 2024-03-18T14:44:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066baf7640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f06640d25a0 con 0x7f066406c730 2024-03-18T14:44:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066baf7640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f06640d2880 con 0x7f06640d1de0 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066a094640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066406c730 0x7f066406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066a094640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066406c730 0x7f066406cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41632/0 (socket says 172.21.15.6:41632) 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066a094640 1 -- 172.21.15.6:0/266580641 learned_addr learned my addr 172.21.15.6:0/266580641 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f0669092640 1 --2- 172.21.15.6:0/266580641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06640d1de0 0x7f06640d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066a094640 1 -- 172.21.15.6:0/266580641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06640d1de0 msgr2=0x7f06640d3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066a094640 1 --2- 172.21.15.6:0/266580641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06640d1de0 0x7f06640d3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066a094640 1 -- 172.21.15.6:0/266580641 >> v1:172.21.15.92:6789/0 conn(0x7f066415e0b0 legacy=0x7f06640b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f066a094640 1 -- 172.21.15.6:0/266580641 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06640d2b60 con 0x7f066406c730 2024-03-18T14:44:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.791+0000 7f0669092640 1 --2- 172.21.15.6:0/266580641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06640d1de0 0x7f06640d3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:22.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.792+0000 7f066a094640 1 --2- 172.21.15.6:0/266580641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f066406cbe0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f065c009e50 tx=0x7f065c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f417260167278d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.792+0000 7f0668891640 1 -- 172.21.15.6:0/266580641 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c02cb90 con 0x7f066406c730 2024-03-18T14:44:22.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.792+0000 7f0668891640 1 -- 172.21.15.6:0/266580641 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f065c02cd30 con 0x7f066406c730 2024-03-18T14:44:22.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.792+0000 7f0668891640 1 -- 172.21.15.6:0/266580641 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c034610 con 0x7f066406c730 2024-03-18T14:44:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.792+0000 7f066baf7640 1 -- 172.21.15.6:0/266580641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 msgr2=0x7f066406cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.792+0000 7f066baf7640 1 --2- 172.21.15.6:0/266580641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f066406cbe0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f065c009e50 tx=0x7f065c02bff0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.793+0000 7f066baf7640 1 -- 172.21.15.6:0/266580641 shutdown_connections 2024-03-18T14:44:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.793+0000 7f066baf7640 1 --2- 172.21.15.6:0/266580641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f066406cbe0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.793+0000 7f066baf7640 1 --2- 172.21.15.6:0/266580641 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06640d1de0 0x7f06640d3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.793+0000 7f066baf7640 1 -- 172.21.15.6:0/266580641 >> 172.21.15.6:0/266580641 conn(0x7f0664072ce0 msgr2=0x7f06640730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.793+0000 7f066baf7640 1 -- 172.21.15.6:0/266580641 shutdown_connections 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.793+0000 7f066baf7640 1 -- 172.21.15.6:0/266580641 wait complete. 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 Processor -- start 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 -- start start 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f06640cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06640d1de0 0x7f066414ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f066415e0b0 0x7f066414ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f06640d38a0 con 0x7f066406c730 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f06640b6d00 con 0x7f06640d1de0 2024-03-18T14:44:22.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.794+0000 7f066baf7640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f0664149d70 con 0x7f066415e0b0 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f06640cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f06640cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41648/0 (socket says 172.21.15.6:41648) 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 -- 172.21.15.6:0/3864791224 learned_addr learned my addr 172.21.15.6:0/3864791224 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f066a094640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f066415e0b0 0x7f066414ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 -- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f066415e0b0 msgr2=0x7f066414ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f066415e0b0 0x7f066414ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669092640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06640d1de0 0x7f066414ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 -- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06640d1de0 msgr2=0x7f066414ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06640d1de0 0x7f066414ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 -- 172.21.15.6:0/3864791224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066414a050 con 0x7f066406c730 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f0669893640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f06640cd990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f066000c3c0 tx=0x7f066000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06600052d0 con 0x7f066406c730 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0660005490 con 0x7f066406c730 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066414a330 con 0x7f066406c730 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660007260 con 0x7f066406c730 2024-03-18T14:44:22.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.795+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066414a8f0 con 0x7f066406c730 2024-03-18T14:44:22.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.796+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f066408afe0 con 0x7f066406c730 2024-03-18T14:44:22.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0660007400 con 0x7f066406c730 2024-03-18T14:44:22.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f065a7fc640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f064806bbd0 0x7f064806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f0660089d50 con 0x7f066406c730 2024-03-18T14:44:22.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f065a7fc640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7f0648072070 0x7f06480744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 --> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0660009e50 con 0x7f0648072070 2024-03-18T14:44:22.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f066008c450 con 0x7f066406c730 2024-03-18T14:44:22.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f0669092640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f064806bbd0 0x7f064806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.797+0000 7f066a094640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7f0648072070 0x7f06480744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.798+0000 7f0669092640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f064806bbd0 0x7f064806e090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f06640b27d0 tx=0x7f0654008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.798+0000 7f066a094640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7f0648072070 0x7f06480744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.799+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== osd.4 v2:172.21.15.92:6816/3636494559 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0660009e50 con 0x7f0648072070 2024-03-18T14:44:22.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.812+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 --> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f066404f620 con 0x7f0648072070 2024-03-18T14:44:22.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.812+0000 7f065a7fc640 1 -- 172.21.15.6:0/3864791224 <== osd.4 v2:172.21.15.92:6816/3636494559 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f066404f620 con 0x7f0648072070 2024-03-18T14:44:22.814 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:22.814 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:22.814 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7f0648072070 msgr2=0x7f06480744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6816/3636494559,v1:172.21.15.92:6817/3636494559] conn(0x7f0648072070 0x7f06480744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f064806bbd0 msgr2=0x7f064806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f064806bbd0 0x7f064806e090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f06640b27d0 tx=0x7f0654008040 comp rx=0 tx=0).stop 2024-03-18T14:44:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 msgr2=0x7f06640cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 --2- 172.21.15.6:0/3864791224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066406c730 0x7f06640cd990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f066000c3c0 tx=0x7f066000c880 comp rx=0 tx=0).stop 2024-03-18T14:44:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066a094640 1 -- 172.21.15.6:0/3864791224 reap_dead start 2024-03-18T14:44:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 shutdown_connections 2024-03-18T14:44:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 >> 172.21.15.6:0/3864791224 conn(0x7f0664072ce0 msgr2=0x7f06640b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 shutdown_connections 2024-03-18T14:44:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.813+0000 7f066baf7640 1 -- 172.21.15.6:0/3864791224 wait complete. 2024-03-18T14:44:22.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-18T14:44:22.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.925+0000 7fde4cc19640 1 Processor -- start 2024-03-18T14:44:22.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.925+0000 7fde4cc19640 1 -- start start 2024-03-18T14:44:22.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.925+0000 7fde4cc19640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fde4815e120 0x7fde480cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.925+0000 7fde4cc19640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde48162700 0x7fde4806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.925+0000 7fde4cc19640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde480b20a0 con 0x7fde480b1af0 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.925+0000 7fde4cc19640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fde480b2380 con 0x7fde48162700 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.925+0000 7fde4cc19640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fde48161c20 con 0x7fde4815e120 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde3ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde48162700 0x7fde4806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde3ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde48162700 0x7fde4806b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41650/0 (socket says 172.21.15.6:41650) 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde3ffff640 1 -- 172.21.15.6:0/3841255197 learned_addr learned my addr 172.21.15.6:0/3841255197 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde4659c640 1 --2- 172.21.15.6:0/3841255197 >> v2:172.21.15.92:3300/0 conn(0x7fde4815e120 0x7fde480cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde3ffff640 1 -- 172.21.15.6:0/3841255197 >> v2:172.21.15.92:3300/0 conn(0x7fde4815e120 msgr2=0x7fde480cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde3ffff640 1 --2- 172.21.15.6:0/3841255197 >> v2:172.21.15.92:3300/0 conn(0x7fde4815e120 0x7fde480cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde3ffff640 1 -- 172.21.15.6:0/3841255197 >> v1:172.21.15.92:6789/0 conn(0x7fde480b1af0 legacy=0x7fde480d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde3ffff640 1 -- 172.21.15.6:0/3841255197 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde48161f00 con 0x7fde48162700 2024-03-18T14:44:22.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.926+0000 7fde4659c640 1 --2- 172.21.15.6:0/3841255197 >> v2:172.21.15.92:3300/0 conn(0x7fde4815e120 0x7fde480cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:22.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde3ffff640 1 --2- 172.21.15.6:0/3841255197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde48162700 0x7fde4806b1c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fde480d0980 tx=0x7fde3002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67bb835370696349 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde45d9b640 1 -- 172.21.15.6:0/3841255197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3002f850 con 0x7fde48162700 2024-03-18T14:44:22.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde45d9b640 1 -- 172.21.15.6:0/3841255197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3002f9f0 con 0x7fde48162700 2024-03-18T14:44:22.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde45d9b640 1 -- 172.21.15.6:0/3841255197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3002fdc0 con 0x7fde48162700 2024-03-18T14:44:22.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 -- 172.21.15.6:0/3841255197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde48162700 msgr2=0x7fde4806b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3841255197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde48162700 0x7fde4806b1c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fde480d0980 tx=0x7fde3002ba30 comp rx=0 tx=0).stop 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 -- 172.21.15.6:0/3841255197 shutdown_connections 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3841255197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde48162700 0x7fde4806b1c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3841255197 >> v2:172.21.15.92:3300/0 conn(0x7fde4815e120 0x7fde480cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 -- 172.21.15.6:0/3841255197 >> 172.21.15.6:0/3841255197 conn(0x7fde48099a50 msgr2=0x7fde48099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 -- 172.21.15.6:0/3841255197 shutdown_connections 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.927+0000 7fde4cc19640 1 -- 172.21.15.6:0/3841255197 wait complete. 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 Processor -- start 2024-03-18T14:44:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 -- start start 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde480b1af0 0x7fde481f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde48162700 0x7fde4815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde48158730 0x7fde48158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fde4806b700 con 0x7fde480b1af0 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fde480ce360 con 0x7fde48158730 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4cc19640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fde4806b880 con 0x7fde48162700 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.929+0000 7fde4659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde480b1af0 0x7fde481f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde480b1af0 0x7fde481f6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41656/0 (socket says 172.21.15.6:41656) 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 -- 172.21.15.6:0/3592408750 learned_addr learned my addr 172.21.15.6:0/3592408750 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde46d9d640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde48158730 0x7fde48158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde3ffff640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde48162700 0x7fde4815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 -- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde48162700 msgr2=0x7fde4815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde48162700 0x7fde4815eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 -- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde48158730 msgr2=0x7fde48158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde48158730 0x7fde48158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 -- 172.21.15.6:0/3592408750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde481f7100 con 0x7fde480b1af0 2024-03-18T14:44:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4659c640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde480b1af0 0x7fde481f6bc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e810 tx=0x7fde3800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde3effd640 1 -- 172.21.15.6:0/3592408750 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3800f040 con 0x7fde480b1af0 2024-03-18T14:44:22.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde3effd640 1 -- 172.21.15.6:0/3592408750 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde38004030 con 0x7fde480b1af0 2024-03-18T14:44:22.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde481602b0 con 0x7fde480b1af0 2024-03-18T14:44:22.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde3effd640 1 -- 172.21.15.6:0/3592408750 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38004400 con 0x7fde480b1af0 2024-03-18T14:44:22.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.930+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde48160870 con 0x7fde480b1af0 2024-03-18T14:44:22.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.931+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde4804f5b0 con 0x7fde480b1af0 2024-03-18T14:44:22.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.932+0000 7fde3effd640 1 -- 172.21.15.6:0/3592408750 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fde3800c6d0 con 0x7fde480b1af0 2024-03-18T14:44:22.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.932+0000 7fde3effd640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde2806bbd0 0x7fde2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:22.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.932+0000 7fde3effd640 1 -- 172.21.15.6:0/3592408750 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7fde3808a650 con 0x7fde480b1af0 2024-03-18T14:44:22.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.932+0000 7fde3ffff640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde2806bbd0 0x7fde2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:22.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.933+0000 7fde3ffff640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde2806bbd0 0x7fde2806e090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fde30036000 tx=0x7fde30008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:22.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:22.936+0000 7fde3effd640 1 -- 172.21.15.6:0/3592408750 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fde38017020 con 0x7fde480b1af0 2024-03-18T14:44:23.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.118+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fde48160060 con 0x7fde2806bbd0 2024-03-18T14:44:23.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.119+0000 7fde3effd640 1 -- 172.21.15.6:0/3592408750 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fde48160060 con 0x7fde2806bbd0 2024-03-18T14:44:23.120 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2024-03-18T14:44:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde2806bbd0 msgr2=0x7fde2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde2806bbd0 0x7fde2806e090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fde30036000 tx=0x7fde30008030 comp rx=0 tx=0).stop 2024-03-18T14:44:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde480b1af0 msgr2=0x7fde481f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde480b1af0 0x7fde481f6bc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e810 tx=0x7fde3800ecd0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 shutdown_connections 2024-03-18T14:44:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fde2806bbd0 0x7fde2806e090 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde48158730 0x7fde48158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde48162700 0x7fde4815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 --2- 172.21.15.6:0/3592408750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde480b1af0 0x7fde481f6bc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.120+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 >> 172.21.15.6:0/3592408750 conn(0x7fde48099a50 msgr2=0x7fde480ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.121+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 shutdown_connections 2024-03-18T14:44:23.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.121+0000 7fde4cc19640 1 -- 172.21.15.6:0/3592408750 wait complete. 2024-03-18T14:44:23.132 INFO:tasks.ceph:Scrubbing osd.5 2024-03-18T14:44:23.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:23.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.236+0000 7faf60bcd640 1 Processor -- start 2024-03-18T14:44:23.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.236+0000 7faf60bcd640 1 -- start start 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf60bcd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c06b7a0 0x7faf5c06b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf60bcd640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf5c072220 0x7faf5c0726b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf60bcd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5c0ae6c0 con 0x7faf5c161770 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf60bcd640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7faf5c0acfc0 con 0x7faf5c072220 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf60bcd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faf5c06c8a0 con 0x7faf5c06b7a0 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf5c072220 0x7faf5c0726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5a59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faf5c161770 0x7faf5c0ada10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55514/0 (socket says 172.21.15.6:55514) 2024-03-18T14:44:23.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5ad9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf5c072220 0x7faf5c0726b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41662/0 (socket says 172.21.15.6:41662) 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5a59c640 1 -- 172.21.15.6:0/2860143220 learned_addr learned my addr 172.21.15.6:0/2860143220 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf59d9b640 1 --2- 172.21.15.6:0/2860143220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c06b7a0 0x7faf5c06b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5ad9d640 1 -- 172.21.15.6:0/2860143220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c06b7a0 msgr2=0x7faf5c06b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5ad9d640 1 --2- 172.21.15.6:0/2860143220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c06b7a0 0x7faf5c06b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5ad9d640 1 -- 172.21.15.6:0/2860143220 >> v1:172.21.15.6:6789/0 conn(0x7faf5c161770 legacy=0x7faf5c0ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5ad9d640 1 -- 172.21.15.6:0/2860143220 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c06cb80 con 0x7faf5c072220 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf5959a640 1 -- 172.21.15.6:0/2860143220 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4024114398 0 0) 0x7faf5c0ae6c0 con 0x7faf5c161770 2024-03-18T14:44:23.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.237+0000 7faf59d9b640 1 --2- 172.21.15.6:0/2860143220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c06b7a0 0x7faf5c06b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.238+0000 7faf5ad9d640 1 --2- 172.21.15.6:0/2860143220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c072220 0x7faf5c0726b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009d90 tx=0x7faf4c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f852ead4362a8a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.238+0000 7faf5959a640 1 -- 172.21.15.6:0/2860143220 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c039070 con 0x7faf5c072220 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.238+0000 7faf5959a640 1 -- 172.21.15.6:0/2860143220 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c02cc50 con 0x7faf5c072220 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.238+0000 7faf5959a640 1 -- 172.21.15.6:0/2860143220 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c034440 con 0x7faf5c072220 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 -- 172.21.15.6:0/2860143220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c072220 msgr2=0x7faf5c0726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 --2- 172.21.15.6:0/2860143220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c072220 0x7faf5c0726b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009d90 tx=0x7faf4c02bea0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 -- 172.21.15.6:0/2860143220 shutdown_connections 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 --2- 172.21.15.6:0/2860143220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c072220 0x7faf5c0726b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 --2- 172.21.15.6:0/2860143220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c06b7a0 0x7faf5c06b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 -- 172.21.15.6:0/2860143220 >> 172.21.15.6:0/2860143220 conn(0x7faf5c0d0150 msgr2=0x7faf5c0d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 -- 172.21.15.6:0/2860143220 shutdown_connections 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.239+0000 7faf60bcd640 1 -- 172.21.15.6:0/2860143220 wait complete. 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.240+0000 7faf60bcd640 1 Processor -- start 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.240+0000 7faf60bcd640 1 -- start start 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf60bcd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c06b7a0 0x7faf5c151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf60bcd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c072220 0x7faf5c1f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf60bcd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf5c161770 0x7faf5c0cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf60bcd640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c0ae120 con 0x7faf5c06b7a0 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf60bcd640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c0ad4f0 con 0x7faf5c161770 2024-03-18T14:44:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf60bcd640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7faf5c0ae2a0 con 0x7faf5c072220 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c06b7a0 0x7faf5c151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c06b7a0 0x7faf5c151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41678/0 (socket says 172.21.15.6:41678) 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 -- 172.21.15.6:0/3685053126 learned_addr learned my addr 172.21.15.6:0/3685053126 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf59d9b640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c072220 0x7faf5c1f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5ad9d640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf5c161770 0x7faf5c0cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 -- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c072220 msgr2=0x7faf5c1f7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf5c072220 0x7faf5c1f7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 -- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf5c161770 msgr2=0x7faf5c0cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf5c161770 0x7faf5c0cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 -- 172.21.15.6:0/3685053126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c1e6c40 con 0x7faf5c06b7a0 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5ad9d640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf5c161770 0x7faf5c0cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.241+0000 7faf5a59c640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c06b7a0 0x7faf5c151500 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7faf5400c430 tx=0x7faf5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.242+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf540074e0 con 0x7faf5c06b7a0 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.242+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf540047e0 con 0x7faf5c06b7a0 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.242+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5c1f82a0 con 0x7faf5c06b7a0 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.242+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54004bd0 con 0x7faf5c06b7a0 2024-03-18T14:44:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.242+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5c0c3db0 con 0x7faf5c06b7a0 2024-03-18T14:44:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.242+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf5c0c3c00 con 0x7faf5c06b7a0 2024-03-18T14:44:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.243+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faf54011070 con 0x7faf5c06b7a0 2024-03-18T14:44:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.243+0000 7faf52ffd640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faf3c06bc00 0x7faf3c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.243+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7faf5408a470 con 0x7faf5c06b7a0 2024-03-18T14:44:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.244+0000 7faf52ffd640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7faf3c0720a0 0x7faf3c0744e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.244+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 --> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf54015e70 con 0x7faf3c0720a0 2024-03-18T14:44:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.244+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7faf54091050 con 0x7faf5c06b7a0 2024-03-18T14:44:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.244+0000 7faf59d9b640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faf3c06bc00 0x7faf3c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.244+0000 7faf5ad9d640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7faf3c0720a0 0x7faf3c0744e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.244+0000 7faf59d9b640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faf3c06bc00 0x7faf3c06e0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faf5c161180 tx=0x7faf44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.244+0000 7faf5ad9d640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7faf3c0720a0 0x7faf3c0744e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.245+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== osd.5 v2:172.21.15.92:6808/4136952425 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf54015e70 con 0x7faf3c0720a0 2024-03-18T14:44:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.258+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 --> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faf5c0c6630 con 0x7faf3c0720a0 2024-03-18T14:44:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf52ffd640 1 -- 172.21.15.6:0/3685053126 <== osd.5 v2:172.21.15.92:6808/4136952425 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faf5c0c6630 con 0x7faf3c0720a0 2024-03-18T14:44:23.260 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:23.260 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:23.260 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7faf3c0720a0 msgr2=0x7faf3c0744e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6808/4136952425,v1:172.21.15.92:6809/4136952425] conn(0x7faf3c0720a0 0x7faf3c0744e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faf3c06bc00 msgr2=0x7faf3c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7faf3c06bc00 0x7faf3c06e0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faf5c161180 tx=0x7faf44008040 comp rx=0 tx=0).stop 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c06b7a0 msgr2=0x7faf5c151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 --2- 172.21.15.6:0/3685053126 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf5c06b7a0 0x7faf5c151500 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7faf5400c430 tx=0x7faf5400c8f0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf5ad9d640 1 -- 172.21.15.6:0/3685053126 reap_dead start 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 shutdown_connections 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 >> 172.21.15.6:0/3685053126 conn(0x7faf5c0d0150 msgr2=0x7faf5c0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 shutdown_connections 2024-03-18T14:44:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.259+0000 7faf60bcd640 1 -- 172.21.15.6:0/3685053126 wait complete. 2024-03-18T14:44:23.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-18T14:44:23.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.371+0000 7f10c8ef1640 1 Processor -- start 2024-03-18T14:44:23.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.371+0000 7f10c8ef1640 1 -- start start 2024-03-18T14:44:23.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c8ef1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406c7a0 0x7f10c40d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c8ef1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f10c406cb80 0x7f10c40b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c8ef1640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f10c40b6df0 0x7f10c40b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c8ef1640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f10c40d1f30 con 0x7f10c40b6df0 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c8ef1640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f10c40d2210 con 0x7f10c406cb80 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c8ef1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f10c40d24f0 con 0x7f10c406c7a0 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c2d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f10c40b6df0 0x7f10c40b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c2d9d640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f10c40b6df0 0x7f10c40b7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41684/0 (socket says 172.21.15.6:41684) 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.372+0000 7f10c2d9d640 1 -- 172.21.15.6:0/2722197091 learned_addr learned my addr 172.21.15.6:0/2722197091 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c259c640 1 --2- 172.21.15.6:0/2722197091 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406c7a0 0x7f10c40d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c1d9b640 1 --2- 172.21.15.6:0/2722197091 >> v2:172.21.15.92:3300/0 conn(0x7f10c406cb80 0x7f10c40b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c2d9d640 1 -- 172.21.15.6:0/2722197091 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406c7a0 msgr2=0x7f10c40d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:23.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c2d9d640 1 --2- 172.21.15.6:0/2722197091 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406c7a0 0x7f10c40d2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c2d9d640 1 -- 172.21.15.6:0/2722197091 >> v2:172.21.15.92:3300/0 conn(0x7f10c406cb80 msgr2=0x7f10c40b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:23.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c2d9d640 1 --2- 172.21.15.6:0/2722197091 >> v2:172.21.15.92:3300/0 conn(0x7f10c406cb80 0x7f10c40b95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c2d9d640 1 -- 172.21.15.6:0/2722197091 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c40d27d0 con 0x7f10c40b6df0 2024-03-18T14:44:23.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c259c640 1 --2- 172.21.15.6:0/2722197091 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406c7a0 0x7f10c40d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T14:44:23.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.373+0000 7f10c2d9d640 1 --2- 172.21.15.6:0/2722197091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c40b7250 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f10c40b6830 tx=0x7f10b402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c2eff733276d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c159a640 1 -- 172.21.15.6:0/2722197091 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b402cb00 con 0x7f10c40b6df0 2024-03-18T14:44:23.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c159a640 1 -- 172.21.15.6:0/2722197091 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b402cca0 con 0x7f10c40b6df0 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c159a640 1 -- 172.21.15.6:0/2722197091 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4034610 con 0x7f10c40b6df0 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 -- 172.21.15.6:0/2722197091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 msgr2=0x7f10c40b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/2722197091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c40b7250 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f10c40b6830 tx=0x7f10b402bf60 comp rx=0 tx=0).stop 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 -- 172.21.15.6:0/2722197091 shutdown_connections 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/2722197091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c40b7250 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/2722197091 >> v2:172.21.15.92:3300/0 conn(0x7f10c406cb80 0x7f10c40b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/2722197091 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406c7a0 0x7f10c40d2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 -- 172.21.15.6:0/2722197091 >> 172.21.15.6:0/2722197091 conn(0x7f10c4072ce0 msgr2=0x7f10c40730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 -- 172.21.15.6:0/2722197091 shutdown_connections 2024-03-18T14:44:23.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.374+0000 7f10c8ef1640 1 -- 172.21.15.6:0/2722197091 wait complete. 2024-03-18T14:44:23.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 Processor -- start 2024-03-18T14:44:23.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 -- start start 2024-03-18T14:44:23.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10c406c7a0 0x7f10c41f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406cb80 0x7f10c41f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c415e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f10c40b9b20 con 0x7f10c40b6df0 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10c406cfe0 con 0x7f10c406c7a0 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c8ef1640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f10c40b9ca0 con 0x7f10c406cb80 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c2d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c2d9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c415e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41700/0 (socket says 172.21.15.6:41700) 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.376+0000 7f10c2d9d640 1 -- 172.21.15.6:0/3676016654 learned_addr learned my addr 172.21.15.6:0/3676016654 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c1d9b640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406cb80 0x7f10c41f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c259c640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10c406c7a0 0x7f10c41f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c2d9d640 1 -- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406cb80 msgr2=0x7f10c41f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c2d9d640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406cb80 0x7f10c41f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c2d9d640 1 -- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10c406c7a0 msgr2=0x7f10c41f36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c2d9d640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10c406c7a0 0x7f10c41f36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c2d9d640 1 -- 172.21.15.6:0/3676016654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c415e6d0 con 0x7f10c40b6df0 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c2d9d640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c415e190 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f10b4036040 tx=0x7f10b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10baffd640 1 -- 172.21.15.6:0/3676016654 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4044020 con 0x7f10c40b6df0 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10baffd640 1 -- 172.21.15.6:0/3676016654 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b402ce50 con 0x7f10c40b6df0 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c415e970 con 0x7f10c40b6df0 2024-03-18T14:44:23.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10baffd640 1 -- 172.21.15.6:0/3676016654 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4009510 con 0x7f10c40b6df0 2024-03-18T14:44:23.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.377+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c415ef30 con 0x7f10c40b6df0 2024-03-18T14:44:23.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.378+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c404f5b0 con 0x7f10c40b6df0 2024-03-18T14:44:23.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.378+0000 7f10baffd640 1 -- 172.21.15.6:0/3676016654 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f10b4047020 con 0x7f10c40b6df0 2024-03-18T14:44:23.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.379+0000 7f10baffd640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10a406bbd0 0x7f10a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.379+0000 7f10baffd640 1 -- 172.21.15.6:0/3676016654 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f10b40ad1d0 con 0x7f10c40b6df0 2024-03-18T14:44:23.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.379+0000 7f10c259c640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10a406bbd0 0x7f10a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.379+0000 7f10c259c640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10a406bbd0 0x7f10a406e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f10bc006fd0 tx=0x7f10bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.383+0000 7f10baffd640 1 -- 172.21.15.6:0/3676016654 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f10b404e030 con 0x7f10c40b6df0 2024-03-18T14:44:23.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.565+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f10c40d4610 con 0x7f10a406bbd0 2024-03-18T14:44:23.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.566+0000 7f10baffd640 1 -- 172.21.15.6:0/3676016654 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f10c40d4610 con 0x7f10a406bbd0 2024-03-18T14:44:23.567 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2024-03-18T14:44:23.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10a406bbd0 msgr2=0x7f10a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10a406bbd0 0x7f10a406e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f10bc006fd0 tx=0x7f10bc008040 comp rx=0 tx=0).stop 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 msgr2=0x7f10c415e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c415e190 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f10b4036040 tx=0x7f10b4004a40 comp rx=0 tx=0).stop 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 shutdown_connections 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f10a406bbd0 0x7f10a406e090 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10c40b6df0 0x7f10c415e190 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10c406cb80 0x7f10c41f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 --2- 172.21.15.6:0/3676016654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10c406c7a0 0x7f10c41f36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 >> 172.21.15.6:0/3676016654 conn(0x7f10c4072ce0 msgr2=0x7f10c40b8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 shutdown_connections 2024-03-18T14:44:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.568+0000 7f10c8ef1640 1 -- 172.21.15.6:0/3676016654 wait complete. 2024-03-18T14:44:23.580 INFO:tasks.ceph:Scrubbing osd.6 2024-03-18T14:44:23.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:23.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.684+0000 7ff3222e8640 1 Processor -- start 2024-03-18T14:44:23.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.684+0000 7ff3222e8640 1 -- start start 2024-03-18T14:44:23.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff3222e8640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff31c0d1aa0 0x7ff31c0d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff3222e8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c0cf6e0 0x7ff31c0cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff3222e8640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff31c0b1400 con 0x7ff31c1599b0 2024-03-18T14:44:23.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff3222e8640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff31c0b17f0 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff3222e8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff31c0cf200 con 0x7ff31c0cf6e0 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff31b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff31c0d1aa0 0x7ff31c0d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff31b7fe640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff31c0d1aa0 0x7ff31c0d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37702/0 (socket says 172.21.15.6:37702) 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.685+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2963355725 learned_addr learned my addr 172.21.15.6:0/2963355725 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.686+0000 7ff320885640 1 --2- 172.21.15.6:0/2963355725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c0cf6e0 0x7ff31c0cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.686+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2963355725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c0cf6e0 msgr2=0x7ff31c0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.686+0000 7ff31b7fe640 1 --2- 172.21.15.6:0/2963355725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c0cf6e0 0x7ff31c0cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.686+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2963355725 >> v1:172.21.15.92:6789/0 conn(0x7ff31c1599b0 legacy=0x7ff31c0b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.686+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2963355725 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31c0b1a40 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.686+0000 7ff31b7fe640 1 --2- 172.21.15.6:0/2963355725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c0d1f50 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff304009870 tx=0x7ff30402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8081c74d8ebe960 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff31affd640 1 -- 172.21.15.6:0/2963355725 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304034070 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff31affd640 1 -- 172.21.15.6:0/2963355725 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff304030840 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff31affd640 1 -- 172.21.15.6:0/2963355725 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304030c10 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 -- 172.21.15.6:0/2963355725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 msgr2=0x7ff31c0d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 --2- 172.21.15.6:0/2963355725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c0d1f50 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff304009870 tx=0x7ff30402b980 comp rx=0 tx=0).stop 2024-03-18T14:44:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 -- 172.21.15.6:0/2963355725 shutdown_connections 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 --2- 172.21.15.6:0/2963355725 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c0cf6e0 0x7ff31c0cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 --2- 172.21.15.6:0/2963355725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c0d1f50 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 -- 172.21.15.6:0/2963355725 >> 172.21.15.6:0/2963355725 conn(0x7ff31c09ae20 msgr2=0x7ff31c08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 -- 172.21.15.6:0/2963355725 shutdown_connections 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.687+0000 7ff3222e8640 1 -- 172.21.15.6:0/2963355725 wait complete. 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.688+0000 7ff3222e8640 1 Processor -- start 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff3222e8640 1 -- start start 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff3222e8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff31c0cf6e0 0x7ff31c0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff3222e8640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c06da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff3222e8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c1599b0 0x7ff31c06c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff3222e8640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c0d0050 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff3222e8640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c0b2000 con 0x7ff31c0cf6e0 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff3222e8640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7ff31c06c810 con 0x7ff31c1599b0 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff31b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c06da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff31b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c06da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37704/0 (socket says 172.21.15.6:37704) 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.689+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2839571907 learned_addr learned my addr 172.21.15.6:0/2839571907 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:23.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff320885640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c1599b0 0x7ff31c06c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c1599b0 msgr2=0x7ff31c06c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff31bfff640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff31c0cf6e0 0x7ff31c0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff31b7fe640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff31c1599b0 0x7ff31c06c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff31c0cf6e0 msgr2=0x7ff31c0c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff31b7fe640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff31c0cf6e0 0x7ff31c0c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff31b7fe640 1 -- 172.21.15.6:0/2839571907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31c06ca40 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff31b7fe640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c06da40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff304000c00 tx=0x7ff304004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304048070 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff304030640 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff31c06cd20 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304034040 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.690+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff31c15ade0 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.691+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff31c06d240 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff30404b070 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff318ff9640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2fc06bbd0 0x7ff2fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7ff3040ac800 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff318ff9640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7ff2fc072070 0x7ff2fc0744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 --> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff304007e80 con 0x7ff2fc072070 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff31bfff640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2fc06bbd0 0x7ff2fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7ff3040b3050 con 0x7ff31c0d1aa0 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.692+0000 7ff320885640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7ff2fc072070 0x7ff2fc0744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.693+0000 7ff31bfff640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2fc06bbd0 0x7ff2fc06e090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff30c000f60 tx=0x7ff30c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.693+0000 7ff320885640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7ff2fc072070 0x7ff2fc0744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.694+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== osd.6 v2:172.21.15.92:6800/1041453586 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff304007e80 con 0x7ff2fc072070 2024-03-18T14:44:23.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.707+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 --> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff31c0c5ac0 con 0x7ff2fc072070 2024-03-18T14:44:23.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.707+0000 7ff318ff9640 1 -- 172.21.15.6:0/2839571907 <== osd.6 v2:172.21.15.92:6800/1041453586 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff31c0c5ac0 con 0x7ff2fc072070 2024-03-18T14:44:23.709 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:23.709 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:23.709 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7ff2fc072070 msgr2=0x7ff2fc0744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6800/1041453586,v1:172.21.15.92:6801/1041453586] conn(0x7ff2fc072070 0x7ff2fc0744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2fc06bbd0 msgr2=0x7ff2fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7ff2fc06bbd0 0x7ff2fc06e090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff30c000f60 tx=0x7ff30c009040 comp rx=0 tx=0).stop 2024-03-18T14:44:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 msgr2=0x7ff31c06da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 --2- 172.21.15.6:0/2839571907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff31c0d1aa0 0x7ff31c06da40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff304000c00 tx=0x7ff304004730 comp rx=0 tx=0).stop 2024-03-18T14:44:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff320885640 1 -- 172.21.15.6:0/2839571907 reap_dead start 2024-03-18T14:44:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 shutdown_connections 2024-03-18T14:44:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 >> 172.21.15.6:0/2839571907 conn(0x7ff31c09ae20 msgr2=0x7ff31c0cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 shutdown_connections 2024-03-18T14:44:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.708+0000 7ff3222e8640 1 -- 172.21.15.6:0/2839571907 wait complete. 2024-03-18T14:44:23.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-18T14:44:23.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.820+0000 7f96fc6b2640 1 Processor -- start 2024-03-18T14:44:23.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.820+0000 7f96fc6b2640 1 -- start start 2024-03-18T14:44:23.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fc6b2640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f415ad20 0x7f96f415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fc6b2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f40b2870 con 0x7f96f406ead0 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fc6b2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f415b710 con 0x7f96f40cdce0 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fc6b2640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f96f415b9f0 con 0x7f96f415ad20 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fa44e640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96f415ad20 0x7f96f415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fac4f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f96f406ead0 0x7f96f40b2d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58102/0 (socket says 172.21.15.6:58102) 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fac4f640 1 -- 172.21.15.6:0/3171443517 learned_addr learned my addr 172.21.15.6:0/3171443517 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fa44e640 1 -- 172.21.15.6:0/3171443517 >> v1:172.21.15.92:6789/0 conn(0x7f96f40cdce0 legacy=0x7f96f40ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fa44e640 1 -- 172.21.15.6:0/3171443517 >> v1:172.21.15.6:6789/0 conn(0x7f96f406ead0 legacy=0x7f96f40b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96fa44e640 1 -- 172.21.15.6:0/3171443517 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f415bcd0 con 0x7f96f415ad20 2024-03-18T14:44:23.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.821+0000 7f96f944c640 1 -- 172.21.15.6:0/3171443517 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045285093 0 0) 0x7f96f40b2870 con 0x7f96f406ead0 2024-03-18T14:44:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.822+0000 7f96fa44e640 1 --2- 172.21.15.6:0/3171443517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415b100 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0097a0 tx=0x7f96ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e1986f1838a04c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.822+0000 7f96f944c640 1 -- 172.21.15.6:0/3171443517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec035020 con 0x7f96f415ad20 2024-03-18T14:44:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.822+0000 7f96f944c640 1 -- 172.21.15.6:0/3171443517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ec031830 con 0x7f96f415ad20 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.822+0000 7f96f944c640 1 -- 172.21.15.6:0/3171443517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec031c00 con 0x7f96f415ad20 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.823+0000 7f96fc6b2640 1 -- 172.21.15.6:0/3171443517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 msgr2=0x7f96f415b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.823+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/3171443517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415b100 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0097a0 tx=0x7f96ec02b8d0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.823+0000 7f96fc6b2640 1 -- 172.21.15.6:0/3171443517 shutdown_connections 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.823+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/3171443517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415b100 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.823+0000 7f96fc6b2640 1 -- 172.21.15.6:0/3171443517 >> 172.21.15.6:0/3171443517 conn(0x7f96f40b1570 msgr2=0x7f96f40b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.823+0000 7f96fc6b2640 1 -- 172.21.15.6:0/3171443517 shutdown_connections 2024-03-18T14:44:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.823+0000 7f96fc6b2640 1 -- 172.21.15.6:0/3171443517 wait complete. 2024-03-18T14:44:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.824+0000 7f96fc6b2640 1 Processor -- start 2024-03-18T14:44:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.824+0000 7f96fc6b2640 1 -- start start 2024-03-18T14:44:23.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fc6b2640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96f406ead0 0x7f96f41515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fc6b2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96f40cdce0 0x7f96f40ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fc6b2640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fc6b2640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96f40ce2c0 con 0x7f96f415ad20 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fc6b2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f96f40ce140 con 0x7f96f40cdce0 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fc6b2640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f96f406ef20 con 0x7f96f406ead0 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37724/0 (socket says 172.21.15.6:37724) 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 -- 172.21.15.6:0/422833270 learned_addr learned my addr 172.21.15.6:0/422833270 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fa44e640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96f406ead0 0x7f96f41515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96f9c4d640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96f40cdce0 0x7f96f40ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 -- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96f406ead0 msgr2=0x7f96f41515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96f406ead0 0x7f96f41515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 -- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96f40cdce0 msgr2=0x7f96f40ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96f40cdce0 0x7f96f40ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 -- 172.21.15.6:0/422833270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f4144840 con 0x7f96f415ad20 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.825+0000 7f96fac4f640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415d640 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f96f000c7a0 tx=0x7f96f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.826+0000 7f96eaffd640 1 -- 172.21.15.6:0/422833270 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0018070 con 0x7f96f415ad20 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.826+0000 7f96eaffd640 1 -- 172.21.15.6:0/422833270 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f000a640 con 0x7f96f415ad20 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.826+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f4150b10 con 0x7f96f415ad20 2024-03-18T14:44:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.826+0000 7f96eaffd640 1 -- 172.21.15.6:0/422833270 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f000aab0 con 0x7f96f415ad20 2024-03-18T14:44:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.826+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f41510d0 con 0x7f96f415ad20 2024-03-18T14:44:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.827+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f404f5b0 con 0x7f96f415ad20 2024-03-18T14:44:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.827+0000 7f96eaffd640 1 -- 172.21.15.6:0/422833270 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f96f001b070 con 0x7f96f415ad20 2024-03-18T14:44:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.827+0000 7f96eaffd640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f96d806bc00 0x7f96d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.828+0000 7f96eaffd640 1 -- 172.21.15.6:0/422833270 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f96f0089e70 con 0x7f96f415ad20 2024-03-18T14:44:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.828+0000 7f96fa44e640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f96d806bc00 0x7f96d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.828+0000 7f96fa44e640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f96d806bc00 0x7f96d806e0c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f96ec036000 tx=0x7f96ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:23.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:23.832+0000 7f96eaffd640 1 -- 172.21.15.6:0/422833270 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f96f0058480 con 0x7f96f415ad20 2024-03-18T14:44:24.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.014+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f96f4144b20 con 0x7f96d806bc00 2024-03-18T14:44:24.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.015+0000 7f96eaffd640 1 -- 172.21.15.6:0/422833270 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f96f4144b20 con 0x7f96d806bc00 2024-03-18T14:44:24.016 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 6 to deep-scrub 2024-03-18T14:44:24.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.016+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f96d806bc00 msgr2=0x7f96d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.016+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f96d806bc00 0x7f96d806e0c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f96ec036000 tx=0x7f96ec008030 comp rx=0 tx=0).stop 2024-03-18T14:44:24.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.016+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 msgr2=0x7f96f415d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.016+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415d640 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f96f000c7a0 tx=0x7f96f000cc60 comp rx=0 tx=0).stop 2024-03-18T14:44:24.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 shutdown_connections 2024-03-18T14:44:24.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f96d806bc00 0x7f96d806e0c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96f415ad20 0x7f96f415d640 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96f40cdce0 0x7f96f40ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 --2- 172.21.15.6:0/422833270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96f406ead0 0x7f96f41515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 >> 172.21.15.6:0/422833270 conn(0x7f96f40b1570 msgr2=0x7f96f40afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:24.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 shutdown_connections 2024-03-18T14:44:24.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.017+0000 7f96fc6b2640 1 -- 172.21.15.6:0/422833270 wait complete. 2024-03-18T14:44:24.028 INFO:tasks.ceph:Scrubbing osd.7 2024-03-18T14:44:24.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-18T14:44:24.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 Processor -- start 2024-03-18T14:44:24.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 -- start start 2024-03-18T14:44:24.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f977415e000 0x7f97740b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740d1a50 0x7f97740d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f97740ce3b0 0x7f97740ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f97740b28b0 con 0x7f977415e000 2024-03-18T14:44:24.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f97740d23d0 con 0x7f97740ce3b0 2024-03-18T14:44:24.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.133+0000 7f977a5df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f977415e3e0 con 0x7f97740d1a50 2024-03-18T14:44:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f977415e000 0x7f97740b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f977415e000 0x7f97740b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37734/0 (socket says 172.21.15.6:37734) 2024-03-18T14:44:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 -- 172.21.15.6:0/3521017862 learned_addr learned my addr 172.21.15.6:0/3521017862 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9778b7c640 1 --2- 172.21.15.6:0/3521017862 >> v2:172.21.15.92:3300/0 conn(0x7f97740ce3b0 0x7f97740ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 -- 172.21.15.6:0/3521017862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740d1a50 msgr2=0x7f97740d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f97737fe640 1 --2- 172.21.15.6:0/3521017862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740d1a50 0x7f97740d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 --2- 172.21.15.6:0/3521017862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740d1a50 0x7f97740d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 -- 172.21.15.6:0/3521017862 >> v2:172.21.15.92:3300/0 conn(0x7f97740ce3b0 msgr2=0x7f97740ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 --2- 172.21.15.6:0/3521017862 >> v2:172.21.15.92:3300/0 conn(0x7f97740ce3b0 0x7f97740ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9773fff640 1 -- 172.21.15.6:0/3521017862 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97740b1ca0 con 0x7f977415e000 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f9778b7c640 1 --2- 172.21.15.6:0/3521017862 >> v2:172.21.15.92:3300/0 conn(0x7f97740ce3b0 0x7f97740ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.134+0000 7f97737fe640 1 --2- 172.21.15.6:0/3521017862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740d1a50 0x7f97740d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f9773fff640 1 --2- 172.21.15.6:0/3521017862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740b2e00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f97740b2b90 tx=0x7f976402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbdea9a5bf3425f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f9772ffd640 1 -- 172.21.15.6:0/3521017862 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764034070 con 0x7f977415e000 2024-03-18T14:44:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f9772ffd640 1 -- 172.21.15.6:0/3521017862 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9764030850 con 0x7f977415e000 2024-03-18T14:44:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f9772ffd640 1 -- 172.21.15.6:0/3521017862 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764030c20 con 0x7f977415e000 2024-03-18T14:44:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 -- 172.21.15.6:0/3521017862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 msgr2=0x7f97740b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 --2- 172.21.15.6:0/3521017862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740b2e00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f97740b2b90 tx=0x7f976402b990 comp rx=0 tx=0).stop 2024-03-18T14:44:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 -- 172.21.15.6:0/3521017862 shutdown_connections 2024-03-18T14:44:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 --2- 172.21.15.6:0/3521017862 >> v2:172.21.15.92:3300/0 conn(0x7f97740ce3b0 0x7f97740ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 --2- 172.21.15.6:0/3521017862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740d1a50 0x7f97740d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 --2- 172.21.15.6:0/3521017862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740b2e00 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 -- 172.21.15.6:0/3521017862 >> 172.21.15.6:0/3521017862 conn(0x7f977409ae00 msgr2=0x7f977408b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 -- 172.21.15.6:0/3521017862 shutdown_connections 2024-03-18T14:44:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.135+0000 7f977a5df640 1 -- 172.21.15.6:0/3521017862 wait complete. 2024-03-18T14:44:24.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.136+0000 7f977a5df640 1 Processor -- start 2024-03-18T14:44:24.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f977a5df640 1 -- start start 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f977a5df640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740ce3b0 0x7f97741599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f977a5df640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97740d1a50 0x7f97740af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f977a5df640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f977a5df640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f97740b2540 con 0x7f977415e000 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f977a5df640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97740b2260 con 0x7f97740d1a50 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f977a5df640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f97740b1210 con 0x7f97740ce3b0 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f9778b7c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f9778b7c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740afc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37738/0 (socket says 172.21.15.6:37738) 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.137+0000 7f9778b7c640 1 -- 172.21.15.6:0/3253274142 learned_addr learned my addr 172.21.15.6:0/3253274142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9773fff640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740ce3b0 0x7f97741599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9778b7c640 1 -- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740ce3b0 msgr2=0x7f97741599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f97737fe640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97740d1a50 0x7f97740af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9778b7c640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740ce3b0 0x7f97741599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9778b7c640 1 -- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97740d1a50 msgr2=0x7f97740af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9778b7c640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97740d1a50 0x7f97740af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9778b7c640 1 -- 172.21.15.6:0/3253274142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9774159ef0 con 0x7f977415e000 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9773fff640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97740ce3b0 0x7f97741599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f97737fe640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97740d1a50 0x7f97740af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9778b7c640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740afc80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f976c00cef0 tx=0x7f976c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c01a070 con 0x7f977415e000 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f976c00d860 con 0x7f977415e000 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97740b6c80 con 0x7f977415e000 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c00dc50 con 0x7f977415e000 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.138+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97740c9080 con 0x7f977415e000 2024-03-18T14:44:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.139+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f977409fa70 con 0x7f977415e000 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.139+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f976c01d070 con 0x7f977415e000 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.140+0000 7f9770ff9640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f976006bbd0 0x7f976006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.140+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f976c08b170 con 0x7f977415e000 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.140+0000 7f9770ff9640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7f9760072070 0x7f97600744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.140+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 --> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9760074b80 con 0x7f9760072070 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.140+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f976c092050 con 0x7f977415e000 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.140+0000 7f9773fff640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f976006bbd0 0x7f976006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.140+0000 7f97737fe640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7f9760072070 0x7f97600744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.141+0000 7f9773fff640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f976006bbd0 0x7f976006e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9764036000 tx=0x7f976403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.141+0000 7f97737fe640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7f9760072070 0x7f97600744b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.141+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== osd.7 v2:172.21.15.92:6824/1103567419 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9760074b80 con 0x7f9760072070 2024-03-18T14:44:24.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.155+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 --> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f97740b4440 con 0x7f9760072070 2024-03-18T14:44:24.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.155+0000 7f9770ff9640 1 -- 172.21.15.6:0/3253274142 <== osd.7 v2:172.21.15.92:6824/1103567419 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f97740b4440 con 0x7f9760072070 2024-03-18T14:44:24.157 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-18T14:44:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T14:44:24.157 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-18T14:44:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7f9760072070 msgr2=0x7f97600744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6824/1103567419,v1:172.21.15.92:6825/1103567419] conn(0x7f9760072070 0x7f97600744b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f976006bbd0 msgr2=0x7f976006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f976006bbd0 0x7f976006e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9764036000 tx=0x7f976403b040 comp rx=0 tx=0).stop 2024-03-18T14:44:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 msgr2=0x7f97740afc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 --2- 172.21.15.6:0/3253274142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977415e000 0x7f97740afc80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f976c00cef0 tx=0x7f976c00a710 comp rx=0 tx=0).stop 2024-03-18T14:44:24.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f9778b7c640 1 -- 172.21.15.6:0/3253274142 reap_dead start 2024-03-18T14:44:24.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 shutdown_connections 2024-03-18T14:44:24.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 >> 172.21.15.6:0/3253274142 conn(0x7f977409ae00 msgr2=0x7f97740ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:24.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 shutdown_connections 2024-03-18T14:44:24.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.156+0000 7f977a5df640 1 -- 172.21.15.6:0/3253274142 wait complete. 2024-03-18T14:44:24.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-18T14:44:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.268+0000 7f0391027640 1 Processor -- start 2024-03-18T14:44:24.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.268+0000 7f0391027640 1 -- start start 2024-03-18T14:44:24.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f0391027640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 0x7f038c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f0391027640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f038c0ccc80 0x7f038c0c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f0391027640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f038c0cd450 con 0x7f038c15f2a0 2024-03-18T14:44:24.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f0391027640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f038c15f680 con 0x7f038c0ccc80 2024-03-18T14:44:24.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f0391027640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f038c0cd730 con 0x7f038c0d2180 2024-03-18T14:44:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f038ad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f038c15f2a0 0x7f038c0d1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58110/0 (socket says 172.21.15.6:58110) 2024-03-18T14:44:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f038ad9d640 1 -- 172.21.15.6:0/2076179053 learned_addr learned my addr 172.21.15.6:0/2076179053 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f038b59e640 1 --2- 172.21.15.6:0/2076179053 >> v2:172.21.15.92:3300/0 conn(0x7f038c0ccc80 0x7f038c0c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f038a59c640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 0x7f038c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 175864243 0 0) 0x7f038c0cd450 con 0x7f038c15f2a0 2024-03-18T14:44:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.269+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0374003520 con 0x7f038c15f2a0 2024-03-18T14:44:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1758966498 0 0) 0x7f0374003520 con 0x7f038c15f2a0 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 msgr2=0x7f038c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 0x7f038c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 >> v2:172.21.15.92:3300/0 conn(0x7f038c0ccc80 msgr2=0x7f038c0c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 --2- 172.21.15.6:0/2076179053 >> v2:172.21.15.92:3300/0 conn(0x7f038c0ccc80 0x7f038c0c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c0cda10 con 0x7f038c15f2a0 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 734841367 0 0) 0x7f0384001670 con 0x7f038c15f2a0 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 >> v1:172.21.15.6:6789/0 conn(0x7f038c15f2a0 legacy=0x7f038c0d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03740039b0 0x7f0374003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 0x7f0374004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0374004d40 0x7f03740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f038c15f680 con 0x7f03740039b0 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0cd730 con 0x7f0374004d40 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f037400b9c0 con 0x7f038c0d2180 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03740039b0 0x7f0374003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038a59c640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 0x7f0374004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038b59e640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0374004d40 0x7f03740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038ad9d640 1 -- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 msgr2=0x7f0374004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c0d2180 0x7f0374004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038ad9d640 1 -- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0374004d40 msgr2=0x7f03740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0374004d40 0x7f03740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.270+0000 7f038ad9d640 1 -- 172.21.15.6:0/2076179053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0374003520 con 0x7f03740039b0 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.271+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03740039b0 0x7f0374003dd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f038400c290 tx=0x7f0374005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74e540071a34273 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.271+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384007d50 con 0x7f03740039b0 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.271+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0384030070 con 0x7f03740039b0 2024-03-18T14:44:24.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.271+0000 7f0389d9b640 1 -- 172.21.15.6:0/2076179053 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038402cd80 con 0x7f03740039b0 2024-03-18T14:44:24.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.272+0000 7f0391027640 1 -- 172.21.15.6:0/2076179053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03740039b0 msgr2=0x7f0374003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.272+0000 7f0391027640 1 --2- 172.21.15.6:0/2076179053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03740039b0 0x7f0374003dd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f038400c290 tx=0x7f0374005830 comp rx=0 tx=0).stop 2024-03-18T14:44:24.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.272+0000 7f038b59e640 1 -- 172.21.15.6:0/2076179053 reap_dead start 2024-03-18T14:44:24.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.272+0000 7f0391027640 1 -- 172.21.15.6:0/2076179053 shutdown_connections 2024-03-18T14:44:24.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.272+0000 7f0391027640 1 -- 172.21.15.6:0/2076179053 >> 172.21.15.6:0/2076179053 conn(0x7f038c09ae00 msgr2=0x7f038c08b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:24.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.272+0000 7f0391027640 1 -- 172.21.15.6:0/2076179053 shutdown_connections 2024-03-18T14:44:24.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.272+0000 7f0391027640 1 -- 172.21.15.6:0/2076179053 wait complete. 2024-03-18T14:44:24.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.273+0000 7f0391027640 1 Processor -- start 2024-03-18T14:44:24.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.273+0000 7f0391027640 1 -- start start 2024-03-18T14:44:24.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f0391027640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03740039b0 0x7f038c0ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f0391027640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c155070 0x7f038c0cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f0391027640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0c7800 0x7f038c0c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f0391027640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0cd110 con 0x7f038c0c7800 2024-03-18T14:44:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f0391027640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0cdfd0 con 0x7f03740039b0 2024-03-18T14:44:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f0391027640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f038c0cc740 con 0x7f038c155070 2024-03-18T14:44:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0c7800 0x7f038c0c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0c7800 0x7f038c0c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37758/0 (socket says 172.21.15.6:37758) 2024-03-18T14:44:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 -- 172.21.15.6:0/2072290426 learned_addr learned my addr 172.21.15.6:0/2072290426 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03740039b0 0x7f038c0ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038a59c640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c155070 0x7f038c0cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 -- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c155070 msgr2=0x7f038c0cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c155070 0x7f038c0cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 -- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03740039b0 msgr2=0x7f038c0ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03740039b0 0x7f038c0ae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 -- 172.21.15.6:0/2072290426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c0af590 con 0x7f038c0c7800 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03740039b0 0x7f038c0ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:24.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038a59c640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c155070 0x7f038c0cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.274+0000 7f038b59e640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0c7800 0x7f038c0c7c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f037c011bf0 tx=0x7f037c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.275+0000 7f03737fe640 1 -- 172.21.15.6:0/2072290426 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c00fdb0 con 0x7f038c0c7800 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.275+0000 7f03737fe640 1 -- 172.21.15.6:0/2072290426 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037c01d070 con 0x7f038c0c7800 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.275+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f038c0cbb10 con 0x7f038c0c7800 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.275+0000 7f03737fe640 1 -- 172.21.15.6:0/2072290426 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c015670 con 0x7f038c0c7800 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.275+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f038c0aecf0 con 0x7f038c0c7800 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.276+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f038c0c81c0 con 0x7f038c0c7800 2024-03-18T14:44:24.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.276+0000 7f03737fe640 1 -- 172.21.15.6:0/2072290426 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f037c015810 con 0x7f038c0c7800 2024-03-18T14:44:24.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.276+0000 7f03737fe640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f037806bbd0 0x7f037806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.277+0000 7f03737fe640 1 -- 172.21.15.6:0/2072290426 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f037c08b080 con 0x7f038c0c7800 2024-03-18T14:44:24.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.277+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f037806bbd0 0x7f037806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.277+0000 7f038ad9d640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f037806bbd0 0x7f037806e090 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0384034000 tx=0x7f0384007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.281+0000 7f03737fe640 1 -- 172.21.15.6:0/2072290426 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f038c0cbb10 con 0x7f038c0c7800 2024-03-18T14:44:24.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.462+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f038c156b60 con 0x7f037806bbd0 2024-03-18T14:44:24.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.463+0000 7f03737fe640 1 -- 172.21.15.6:0/2072290426 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f038c156b60 con 0x7f037806bbd0 2024-03-18T14:44:24.464 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 7 to deep-scrub 2024-03-18T14:44:24.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f037806bbd0 msgr2=0x7f037806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f037806bbd0 0x7f037806e090 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0384034000 tx=0x7f0384007000 comp rx=0 tx=0).stop 2024-03-18T14:44:24.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0c7800 msgr2=0x7f038c0c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0c7800 0x7f038c0c7c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f037c011bf0 tx=0x7f037c00f580 comp rx=0 tx=0).stop 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 shutdown_connections 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f037806bbd0 0x7f037806e090 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f038c0c7800 0x7f038c0c7c80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f038c155070 0x7f038c0cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 --2- 172.21.15.6:0/2072290426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03740039b0 0x7f038c0ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 >> 172.21.15.6:0/2072290426 conn(0x7f038c09ae00 msgr2=0x7f038c06fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:24.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 shutdown_connections 2024-03-18T14:44:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.465+0000 7f0391027640 1 -- 172.21.15.6:0/2072290426 wait complete. 2024-03-18T14:44:24.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:44:24.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.576+0000 7f5e2f9d3640 1 Processor -- start 2024-03-18T14:44:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2f9d3640 1 -- start start 2024-03-18T14:44:24.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2f9d3640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e2815e020 0x7f5e28162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2f9d3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e280d3210 0x7f5e280d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2f9d3640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e280b9660 con 0x7f5e280ced00 2024-03-18T14:44:24.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2f9d3640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5e280d2270 con 0x7f5e2815e020 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2f9d3640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f5e280ce390 con 0x7f5e280d3210 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2d76f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e2815e020 0x7f5e28162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2d76f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e2815e020 0x7f5e28162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37768/0 (socket says 172.21.15.6:37768) 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.577+0000 7f5e2d76f640 1 -- 172.21.15.6:0/3114658194 learned_addr learned my addr 172.21.15.6:0/3114658194 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.578+0000 7f5e2df70640 1 --2- 172.21.15.6:0/3114658194 >> v2:172.21.15.92:3300/0 conn(0x7f5e280d3210 0x7f5e280d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.578+0000 7f5e2d76f640 1 -- 172.21.15.6:0/3114658194 >> v2:172.21.15.92:3300/0 conn(0x7f5e280d3210 msgr2=0x7f5e280d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.578+0000 7f5e2d76f640 1 --2- 172.21.15.6:0/3114658194 >> v2:172.21.15.92:3300/0 conn(0x7f5e280d3210 0x7f5e280d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.578+0000 7f5e2d76f640 1 -- 172.21.15.6:0/3114658194 >> v1:172.21.15.92:6789/0 conn(0x7f5e280ced00 legacy=0x7f5e280d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.578+0000 7f5e2d76f640 1 -- 172.21.15.6:0/3114658194 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e280b9900 con 0x7f5e2815e020 2024-03-18T14:44:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.578+0000 7f5e2df70640 1 --2- 172.21.15.6:0/3114658194 >> v2:172.21.15.92:3300/0 conn(0x7f5e280d3210 0x7f5e280d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:24.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.578+0000 7f5e2d76f640 1 --2- 172.21.15.6:0/3114658194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2815e020 0x7f5e28162380 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5e280d1450 tx=0x7f5e2402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55d7dff648343a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e1ffff640 1 -- 172.21.15.6:0/3114658194 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2402e070 con 0x7f5e2815e020 2024-03-18T14:44:24.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e1ffff640 1 -- 172.21.15.6:0/3114658194 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e24037070 con 0x7f5e2815e020 2024-03-18T14:44:24.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e1ffff640 1 -- 172.21.15.6:0/3114658194 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24031a00 con 0x7f5e2815e020 2024-03-18T14:44:24.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/3114658194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2815e020 msgr2=0x7f5e28162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/3114658194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2815e020 0x7f5e28162380 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5e280d1450 tx=0x7f5e2402b880 comp rx=0 tx=0).stop 2024-03-18T14:44:24.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/3114658194 shutdown_connections 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/3114658194 >> v2:172.21.15.92:3300/0 conn(0x7f5e280d3210 0x7f5e280d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/3114658194 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e2815e020 0x7f5e28162380 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/3114658194 >> 172.21.15.6:0/3114658194 conn(0x7f5e28072ce0 msgr2=0x7f5e280730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/3114658194 shutdown_connections 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.579+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/3114658194 wait complete. 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.580+0000 7f5e2f9d3640 1 Processor -- start 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2f9d3640 1 -- start start 2024-03-18T14:44:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2f9d3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e280ced00 0x7f5e28160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2f9d3640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e280d3210 0x7f5e28155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2f9d3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e2815e020 0x7f5e280b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2f9d3640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5e280cf2c0 con 0x7f5e280d3210 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2f9d3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5e280ce890 con 0x7f5e280ced00 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2f9d3640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f5e280cf440 con 0x7f5e2815e020 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2cf6e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e280d3210 0x7f5e28155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2cf6e640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e280d3210 0x7f5e28155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37770/0 (socket says 172.21.15.6:37770) 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2cf6e640 1 -- 172.21.15.6:0/1966092263 learned_addr learned my addr 172.21.15.6:0/1966092263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2d76f640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e280ced00 0x7f5e28160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.581+0000 7f5e2df70640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e2815e020 0x7f5e280b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2cf6e640 1 -- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e2815e020 msgr2=0x7f5e280b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2cf6e640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e2815e020 0x7f5e280b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2cf6e640 1 -- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e280ced00 msgr2=0x7f5e28160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2cf6e640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e280ced00 0x7f5e28160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2cf6e640 1 -- 172.21.15.6:0/1966092263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e280b4990 con 0x7f5e280d3210 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2cf6e640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e280d3210 0x7f5e28155830 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800c410 tx=0x7f5e1800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e1dffb640 1 -- 172.21.15.6:0/1966092263 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18017070 con 0x7f5e280d3210 2024-03-18T14:44:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e1dffb640 1 -- 172.21.15.6:0/1966092263 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e180035f0 con 0x7f5e280d3210 2024-03-18T14:44:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e1dffb640 1 -- 172.21.15.6:0/1966092263 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18003a20 con 0x7f5e280d3210 2024-03-18T14:44:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2815efe0 con 0x7f5e280d3210 2024-03-18T14:44:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.582+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e28155d70 con 0x7f5e280d3210 2024-03-18T14:44:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.583+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2808afe0 con 0x7f5e280d3210 2024-03-18T14:44:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.583+0000 7f5e1dffb640 1 -- 172.21.15.6:0/1966092263 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5e1801a070 con 0x7f5e280d3210 2024-03-18T14:44:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.584+0000 7f5e1dffb640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.584+0000 7f5e1dffb640 1 -- 172.21.15.6:0/1966092263 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f5e18089cb0 con 0x7f5e280d3210 2024-03-18T14:44:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.584+0000 7f5e2d76f640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.585+0000 7f5e2d76f640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e24036000 tx=0x7f5e2403d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:24.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.588+0000 7f5e1dffb640 1 -- 172.21.15.6:0/1966092263 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5e180010c0 con 0x7f5e280d3210 2024-03-18T14:44:24.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.766+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e2804f620 con 0x7f5e0c06bbd0 2024-03-18T14:44:24.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.767+0000 7f5e1dffb640 1 -- 172.21.15.6:0/1966092263 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13395 (secure 0 0 0) 0x7f5e2804f620 con 0x7f5e0c06bbd0 2024-03-18T14:44:24.769 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:24.769 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e0c06bbd0 msgr2=0x7f5e0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e24036000 tx=0x7f5e2403d040 comp rx=0 tx=0).stop 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e280d3210 msgr2=0x7f5e28155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e280d3210 0x7f5e28155830 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800c410 tx=0x7f5e1800c8d0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 shutdown_connections 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e2815e020 0x7f5e280b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e280d3210 0x7f5e28155830 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 --2- 172.21.15.6:0/1966092263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e280ced00 0x7f5e28160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:24.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 >> 172.21.15.6:0/1966092263 conn(0x7f5e28072ce0 msgr2=0x7f5e280d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:24.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 shutdown_connections 2024-03-18T14:44:24.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:24.769+0000 7f5e2f9d3640 1 -- 172.21.15.6:0/1966092263 wait complete. 2024-03-18T14:44:24.781 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-03-18T14:44:23.418298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222944,"kb_used_data":6760,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751776,"statfs":{"total":773094113280,"available":772865818624,"internally_reserved":0,"allocated":6922240,"data_stored":3764528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17096,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-70,"num_write_kb":-41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001662"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-18T14:44:17.198468+0000","last_change":"2024-03-18T14:41:25.748639+0000","last_active":"2024-03-18T14:44:17.198468+0000","last_peered":"2024-03-18T14:44:17.198468+0000","last_clean":"2024-03-18T14:44:17.198468+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:44:17.198468+0000","last_undegraded":"2024-03-18T14:44:17.198468+0000","last_fullsized":"2024-03-18T14:44:17.198468+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T14:41:24.402944+0000","last_clean_scrub_stamp":"2024-03-18T14:41:24.402944+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:32.327590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":913342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1310720,"data_stored":913342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:44:24.784 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-18T14:41:24.402944+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=14, tm_min=41, tm_sec=24, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=14, tm_min=44, tm_sec=19, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T14:44:24.784 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-18T14:44:35.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:35.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:44.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T14:44:44.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.898+0000 7fd56de36640 1 Processor -- start 2024-03-18T14:44:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.898+0000 7fd56de36640 1 -- start start 2024-03-18T14:44:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd56de36640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd5680b17b0 0x7fd5680b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd56de36640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5680d0260 0x7fd5680d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd56de36640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5680cfd80 con 0x7fd56806cce0 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd56de36640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd5680b2df0 con 0x7fd5680d0260 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd56de36640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7fd5680b2270 con 0x7fd5680b17b0 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd567fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5680d0260 0x7fd5680d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd567fff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5680d0260 0x7fd5680d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45814/0 (socket says 172.21.15.6:45814) 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd567fff640 1 -- 172.21.15.6:0/526154197 learned_addr learned my addr 172.21.15.6:0/526154197 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd5677fe640 1 --1- 172.21.15.6:0/526154197 >> v1:172.21.15.6:6789/0 conn(0x7fd56806cce0 0x7fd56806d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38646/0 (socket says 172.21.15.6:38646) 2024-03-18T14:44:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd566ffd640 1 --2- 172.21.15.6:0/526154197 >> v2:172.21.15.92:3300/0 conn(0x7fd5680b17b0 0x7fd5680b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd567fff640 1 -- 172.21.15.6:0/526154197 >> v2:172.21.15.92:3300/0 conn(0x7fd5680b17b0 msgr2=0x7fd5680b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd567fff640 1 --2- 172.21.15.6:0/526154197 >> v2:172.21.15.92:3300/0 conn(0x7fd5680b17b0 0x7fd5680b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd567fff640 1 -- 172.21.15.6:0/526154197 >> v1:172.21.15.6:6789/0 conn(0x7fd56806cce0 legacy=0x7fd56806d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd567fff640 1 -- 172.21.15.6:0/526154197 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5680b2550 con 0x7fd5680d0260 2024-03-18T14:44:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.899+0000 7fd5667fc640 1 -- 172.21.15.6:0/526154197 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3897166252 0 0) 0x7fd5680cfd80 con 0x7fd56806cce0 2024-03-18T14:44:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.900+0000 7fd567fff640 1 --2- 172.21.15.6:0/526154197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5680d0260 0x7fd5680d06d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd56806fb40 tx=0x7fd56002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7444fb98e8e5502 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.900+0000 7fd5667fc640 1 -- 172.21.15.6:0/526154197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56002ca80 con 0x7fd5680d0260 2024-03-18T14:44:44.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.900+0000 7fd5667fc640 1 -- 172.21.15.6:0/526154197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56002cc20 con 0x7fd5680d0260 2024-03-18T14:44:44.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.900+0000 7fd5667fc640 1 -- 172.21.15.6:0/526154197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5600343f0 con 0x7fd5680d0260 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 -- 172.21.15.6:0/526154197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5680d0260 msgr2=0x7fd5680d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 --2- 172.21.15.6:0/526154197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5680d0260 0x7fd5680d06d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd56806fb40 tx=0x7fd56002bee0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 -- 172.21.15.6:0/526154197 shutdown_connections 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 --2- 172.21.15.6:0/526154197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5680d0260 0x7fd5680d06d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 --2- 172.21.15.6:0/526154197 >> v2:172.21.15.92:3300/0 conn(0x7fd5680b17b0 0x7fd5680b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 -- 172.21.15.6:0/526154197 >> 172.21.15.6:0/526154197 conn(0x7fd56808b130 msgr2=0x7fd56808b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 -- 172.21.15.6:0/526154197 shutdown_connections 2024-03-18T14:44:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.901+0000 7fd56de36640 1 -- 172.21.15.6:0/526154197 wait complete. 2024-03-18T14:44:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.902+0000 7fd56de36640 1 Processor -- start 2024-03-18T14:44:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.902+0000 7fd56de36640 1 -- start start 2024-03-18T14:44:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd56de36640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd56806cce0 0x7fd568160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd56de36640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5680b17b0 0x7fd5680af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd56de36640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5680d0260 0x7fd56815e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd56de36640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd56815e330 con 0x7fd56806cce0 2024-03-18T14:44:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd56de36640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd5680b2b10 con 0x7fd5680b17b0 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd56de36640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7fd56806c690 con 0x7fd5680d0260 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd56806cce0 0x7fd568160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd56806cce0 0x7fd568160f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45822/0 (socket says 172.21.15.6:45822) 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 -- 172.21.15.6:0/2131164417 learned_addr learned my addr 172.21.15.6:0/2131164417 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd567fff640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5680d0260 0x7fd56815e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd566ffd640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5680b17b0 0x7fd5680af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 -- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5680d0260 msgr2=0x7fd56815e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5680d0260 0x7fd56815e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 -- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5680b17b0 msgr2=0x7fd5680af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5680b17b0 0x7fd5680af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.903+0000 7fd5677fe640 1 -- 172.21.15.6:0/2131164417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5681614d0 con 0x7fd56806cce0 2024-03-18T14:44:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.904+0000 7fd5677fe640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd56806cce0 0x7fd568160f90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c430 tx=0x7fd55800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.904+0000 7fd54ffff640 1 -- 172.21.15.6:0/2131164417 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5580074e0 con 0x7fd56806cce0 2024-03-18T14:44:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.904+0000 7fd54ffff640 1 -- 172.21.15.6:0/2131164417 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5580047e0 con 0x7fd56806cce0 2024-03-18T14:44:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.904+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5680afa10 con 0x7fd56806cce0 2024-03-18T14:44:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.904+0000 7fd54ffff640 1 -- 172.21.15.6:0/2131164417 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558004bd0 con 0x7fd56806cce0 2024-03-18T14:44:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.904+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5680affd0 con 0x7fd56806cce0 2024-03-18T14:44:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.905+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd56809fa80 con 0x7fd56806cce0 2024-03-18T14:44:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.905+0000 7fd54ffff640 1 -- 172.21.15.6:0/2131164417 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd558011070 con 0x7fd56806cce0 2024-03-18T14:44:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.906+0000 7fd54ffff640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd55406bbd0 0x7fd55406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.906+0000 7fd54ffff640 1 -- 172.21.15.6:0/2131164417 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7fd55808a5c0 con 0x7fd56806cce0 2024-03-18T14:44:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.906+0000 7fd566ffd640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd55406bbd0 0x7fd55406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.907+0000 7fd566ffd640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd55406bbd0 0x7fd55406e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd56815c530 tx=0x7fd55c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:44.910+0000 7fd54ffff640 1 -- 172.21.15.6:0/2131164417 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd558059220 con 0x7fd56806cce0 2024-03-18T14:44:45.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.090+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 --> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd56806cac0 con 0x7fd55406bbd0 2024-03-18T14:44:45.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.091+0000 7fd54ffff640 1 -- 172.21.15.6:0/2131164417 <== mgr.4102 v2:172.21.15.92:6832/4016547990 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7fd56806cac0 con 0x7fd55406bbd0 2024-03-18T14:44:45.093 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:45.093 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd55406bbd0 msgr2=0x7fd55406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd55406bbd0 0x7fd55406e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd56815c530 tx=0x7fd55c008040 comp rx=0 tx=0).stop 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd56806cce0 msgr2=0x7fd568160f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd56806cce0 0x7fd568160f90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c430 tx=0x7fd55800c8f0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 shutdown_connections 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7fd55406bbd0 0x7fd55406e090 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5680d0260 0x7fd56815e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5680b17b0 0x7fd5680af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 --2- 172.21.15.6:0/2131164417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd56806cce0 0x7fd568160f90 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 >> 172.21.15.6:0/2131164417 conn(0x7fd56808b130 msgr2=0x7fd5680ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:45.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 shutdown_connections 2024-03-18T14:44:45.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.093+0000 7fd56de36640 1 -- 172.21.15.6:0/2131164417 wait complete. 2024-03-18T14:44:45.105 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-03-18T14:44:43.422396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222936,"kb_used_data":6760,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751784,"statfs":{"total":773094113280,"available":772865826816,"internally_reserved":0,"allocated":6922240,"data_stored":3764528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002504"},"pg_stats":[{"pgid":"1.0","version":"59'119","reported_seq":250,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-18T14:44:25.069730+0000","last_change":"2024-03-18T14:44:25.069763+0000","last_active":"2024-03-18T14:44:25.069730+0000","last_peered":"2024-03-18T14:44:25.069730+0000","last_clean":"2024-03-18T14:44:25.069730+0000","last_became_active":"2024-03-18T14:41:25.747732+0000","last_became_peered":"2024-03-18T14:41:25.747732+0000","last_unstale":"2024-03-18T14:44:25.069730+0000","last_undegraded":"2024-03-18T14:44:25.069730+0000","last_fullsized":"2024-03-18T14:44:25.069730+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'119","last_scrub_stamp":"2024-03-18T14:44:25.069648+0000","last_deep_scrub":"59'119","last_deep_scrub_stamp":"2024-03-18T14:44:25.069648+0000","last_clean_scrub_stamp":"2024-03-18T14:44:25.069648+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:25.500324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":913342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":322974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1310720,"data_stored":913342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T14:44:45.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-18T14:44:45.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.206+0000 7f9904fbc640 1 Processor -- start 2024-03-18T14:44:45.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.206+0000 7f9904fbc640 1 -- start start 2024-03-18T14:44:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f9904fbc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f990015e030 0x7f99000d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f9904fbc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99000b65a0 con 0x7f99000d0450 2024-03-18T14:44:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f9904fbc640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99000d2fd0 con 0x7f99000b6140 2024-03-18T14:44:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f9904fbc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f990015e410 con 0x7f990015e030 2024-03-18T14:44:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f98fe59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f990015e030 0x7f99000d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f98fed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f99000d0450 0x7f99000d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38650/0 (socket says 172.21.15.6:38650) 2024-03-18T14:44:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f98fe59c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f990015e030 0x7f99000d3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45836/0 (socket says 172.21.15.6:45836) 2024-03-18T14:44:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.207+0000 7f98fed9d640 1 -- 172.21.15.6:0/1076485528 learned_addr learned my addr 172.21.15.6:0/1076485528 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-18T14:44:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.208+0000 7f98fe59c640 1 -- 172.21.15.6:0/1076485528 >> v1:172.21.15.92:6789/0 conn(0x7f99000b6140 legacy=0x7f99000ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.208+0000 7f98fe59c640 1 -- 172.21.15.6:0/1076485528 >> v1:172.21.15.6:6789/0 conn(0x7f99000d0450 legacy=0x7f99000d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.208+0000 7f98fe59c640 1 -- 172.21.15.6:0/1076485528 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99000d3290 con 0x7f990015e030 2024-03-18T14:44:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.208+0000 7f98fe59c640 1 --2- 172.21.15.6:0/1076485528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f99000d3570 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f98f80097a0 tx=0x7f98f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82bd5612ae06362 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.208+0000 7f98fd59a640 1 -- 172.21.15.6:0/1076485528 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8035020 con 0x7f990015e030 2024-03-18T14:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.208+0000 7f98fd59a640 1 -- 172.21.15.6:0/1076485528 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f8031830 con 0x7f990015e030 2024-03-18T14:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f98fd59a640 1 -- 172.21.15.6:0/1076485528 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8031c00 con 0x7f990015e030 2024-03-18T14:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f9904fbc640 1 -- 172.21.15.6:0/1076485528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 msgr2=0x7f99000d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1076485528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f99000d3570 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f98f80097a0 tx=0x7f98f802b8d0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f9904fbc640 1 -- 172.21.15.6:0/1076485528 shutdown_connections 2024-03-18T14:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1076485528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f99000d3570 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f9904fbc640 1 -- 172.21.15.6:0/1076485528 >> 172.21.15.6:0/1076485528 conn(0x7f990006c8c0 msgr2=0x7f990006ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f9904fbc640 1 -- 172.21.15.6:0/1076485528 shutdown_connections 2024-03-18T14:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.209+0000 7f9904fbc640 1 -- 172.21.15.6:0/1076485528 wait complete. 2024-03-18T14:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.210+0000 7f9904fbc640 1 Processor -- start 2024-03-18T14:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.210+0000 7f9904fbc640 1 -- start start 2024-03-18T14:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f9904fbc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99000b6140 0x7f9900160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f9904fbc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99000d0450 0x7f9900158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f9904fbc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f9900159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f9904fbc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f99000ce830 con 0x7f990015e030 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f9904fbc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f99000ce300 con 0x7f99000d0450 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f9904fbc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f9900161a90 con 0x7f99000b6140 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f98fed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f9900159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f98fed9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f9900159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45842/0 (socket says 172.21.15.6:45842) 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f98fed9d640 1 -- 172.21.15.6:0/1383529127 learned_addr learned my addr 172.21.15.6:0/1383529127 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f98fed9d640 1 -- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99000b6140 msgr2=0x7f9900160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f98fe59c640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99000b6140 0x7f9900160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.211+0000 7f98fdd9b640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99000d0450 0x7f9900158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98fed9d640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99000b6140 0x7f9900160ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98fed9d640 1 -- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99000d0450 msgr2=0x7f9900158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98fed9d640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99000d0450 0x7f9900158d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98fed9d640 1 -- 172.21.15.6:0/1383529127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9900155df0 con 0x7f990015e030 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98fed9d640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f9900159280 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f98f000cbf0 tx=0x7f98f000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0017070 con 0x7f990015e030 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f000ad40 con 0x7f990015e030 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99001560d0 con 0x7f990015e030 2024-03-18T14:44:45.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0012600 con 0x7f990015e030 2024-03-18T14:44:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.212+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9900156690 con 0x7f990015e030 2024-03-18T14:44:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.213+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99000751b0 con 0x7f990015e030 2024-03-18T14:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.213+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f98f001a070 con 0x7f990015e030 2024-03-18T14:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.214+0000 7f98eeffd640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98e006bc00 0x7f98e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.214+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f98f008a530 con 0x7f990015e030 2024-03-18T14:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.214+0000 7f98fe59c640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98e006bc00 0x7f98e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.215+0000 7f98fe59c640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98e006bc00 0x7f98e006e0c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f98f8036000 tx=0x7f98f803c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.218+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f98f0058bd0 con 0x7f990015e030 2024-03-18T14:44:45.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.399+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f99000684d0 con 0x7f990015e030 2024-03-18T14:44:45.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.402+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f98f005cc70 con 0x7f990015e030 2024-03-18T14:44:45.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.403+0000 7f98eeffd640 1 -- 172.21.15.6:0/1383529127 <== mon.0 v2:172.21.15.6:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f98f0061230 con 0x7f990015e030 2024-03-18T14:44:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98e006bc00 msgr2=0x7f98e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98e006bc00 0x7f98e006e0c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f98f8036000 tx=0x7f98f803c040 comp rx=0 tx=0).stop 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 msgr2=0x7f9900159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f9900159280 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f98f000cbf0 tx=0x7f98f000a430 comp rx=0 tx=0).stop 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 shutdown_connections 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f98e006bc00 0x7f98e006e0c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f990015e030 0x7f9900159280 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99000d0450 0x7f9900158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 --2- 172.21.15.6:0/1383529127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99000b6140 0x7f9900160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 >> 172.21.15.6:0/1383529127 conn(0x7f990006c8c0 msgr2=0x7f99000b9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:45.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 shutdown_connections 2024-03-18T14:44:45.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.405+0000 7f9904fbc640 1 -- 172.21.15.6:0/1383529127 wait complete. 2024-03-18T14:44:45.417 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T14:44:45.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T14:44:45.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.517+0000 7f3715ccc640 1 Processor -- start 2024-03-18T14:44:45.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.517+0000 7f3715ccc640 1 -- start start 2024-03-18T14:44:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f3715ccc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37100b1740 0x7f37100cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f3715ccc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f37100b1b20 0x7f371006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f3715ccc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f371006b790 0x7f37100caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f3715ccc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f37100ccd40 con 0x7f371006b790 2024-03-18T14:44:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f3715ccc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 -- 0x7f37100cd020 con 0x7f37100b1b20 2024-03-18T14:44:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f3715ccc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f37100cd300 con 0x7f37100b1740 2024-03-18T14:44:45.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f370ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f371006b790 0x7f37100caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f370ffff640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f371006b790 0x7f37100caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45846/0 (socket says 172.21.15.6:45846) 2024-03-18T14:44:45.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.518+0000 7f370ffff640 1 -- 172.21.15.6:0/1902967609 learned_addr learned my addr 172.21.15.6:0/1902967609 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:45.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.519+0000 7f370ffff640 1 -- 172.21.15.6:0/1902967609 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37100b1740 msgr2=0x7f37100cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:45.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.519+0000 7f370ffff640 1 --2- 172.21.15.6:0/1902967609 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37100b1740 0x7f37100cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.519+0000 7f370ffff640 1 -- 172.21.15.6:0/1902967609 >> v2:172.21.15.92:3300/0 conn(0x7f37100b1b20 msgr2=0x7f371006b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T14:44:45.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.519+0000 7f370ffff640 1 --2- 172.21.15.6:0/1902967609 >> v2:172.21.15.92:3300/0 conn(0x7f37100b1b20 0x7f371006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.519+0000 7f370ffff640 1 -- 172.21.15.6:0/1902967609 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37100cd5e0 con 0x7f371006b790 2024-03-18T14:44:45.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.519+0000 7f370ffff640 1 --2- 172.21.15.6:0/1902967609 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f371006b790 0x7f37100caff0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f37100ce490 tx=0x7f370802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be08f7f78591e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:45.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.519+0000 7f370e7fc640 1 -- 172.21.15.6:0/1902967609 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370802cb00 con 0x7f371006b790 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f370e7fc640 1 -- 172.21.15.6:0/1902967609 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f370802cca0 con 0x7f371006b790 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f370e7fc640 1 -- 172.21.15.6:0/1902967609 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3708034630 con 0x7f371006b790 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 -- 172.21.15.6:0/1902967609 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f371006b790 msgr2=0x7f37100caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 --2- 172.21.15.6:0/1902967609 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f371006b790 0x7f37100caff0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f37100ce490 tx=0x7f370802bf60 comp rx=0 tx=0).stop 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 -- 172.21.15.6:0/1902967609 shutdown_connections 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 --2- 172.21.15.6:0/1902967609 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f371006b790 0x7f37100caff0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 --2- 172.21.15.6:0/1902967609 >> v2:172.21.15.92:3300/0 conn(0x7f37100b1b20 0x7f371006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 --2- 172.21.15.6:0/1902967609 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37100b1740 0x7f37100cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 -- 172.21.15.6:0/1902967609 >> 172.21.15.6:0/1902967609 conn(0x7f37100d09a0 msgr2=0x7f37100d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 -- 172.21.15.6:0/1902967609 shutdown_connections 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.520+0000 7f3715ccc640 1 -- 172.21.15.6:0/1902967609 wait complete. 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.521+0000 7f3715ccc640 1 Processor -- start 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.521+0000 7f3715ccc640 1 -- start start 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f3715ccc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371006b790 0x7f37100b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f3715ccc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37100b1740 0x7f37101e9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f3715ccc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37100b1b20 0x7f37101dc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f3715ccc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f37100b1f60 con 0x7f37100b1740 2024-03-18T14:44:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f3715ccc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f371006bc00 con 0x7f37100b1b20 2024-03-18T14:44:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f3715ccc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 -- 0x7f37100c6150 con 0x7f371006b790 2024-03-18T14:44:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f370effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37100b1740 0x7f37101e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f370effd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37100b1740 0x7f37101e9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45854/0 (socket says 172.21.15.6:45854) 2024-03-18T14:44:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f370effd640 1 -- 172.21.15.6:0/125286451 learned_addr learned my addr 172.21.15.6:0/125286451 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-18T14:44:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f370ffff640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37100b1b20 0x7f37101dc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.522+0000 7f370f7fe640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371006b790 0x7f37100b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f370effd640 1 -- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371006b790 msgr2=0x7f37100b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f370effd640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371006b790 0x7f37100b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f370effd640 1 -- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37100b1b20 msgr2=0x7f37101dc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f370effd640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37100b1b20 0x7f37101dc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f370effd640 1 -- 172.21.15.6:0/125286451 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37100c5820 con 0x7f37100b1740 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f370ffff640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37100b1b20 0x7f37101dc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f370effd640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37100b1740 0x7f37101e9580 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f370400b4a0 tx=0x7f370400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f36f7fff640 1 -- 172.21.15.6:0/125286451 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3704014070 con 0x7f37100b1740 2024-03-18T14:44:45.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f36f7fff640 1 -- 172.21.15.6:0/125286451 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3704010610 con 0x7f37100b1740 2024-03-18T14:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37100c5b00 con 0x7f37100b1740 2024-03-18T14:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f36f7fff640 1 -- 172.21.15.6:0/125286451 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3704010a20 con 0x7f37100b1740 2024-03-18T14:44:45.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.523+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37100b4950 con 0x7f37100b1740 2024-03-18T14:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.524+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371004f620 con 0x7f37100b1740 2024-03-18T14:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.524+0000 7f36f7fff640 1 -- 172.21.15.6:0/125286451 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3704017070 con 0x7f37100b1740 2024-03-18T14:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.525+0000 7f36f7fff640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f36fc06bbd0 0x7f36fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T14:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.525+0000 7f36f7fff640 1 -- 172.21.15.6:0/125286451 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5335+0+0 (secure 0 0 0) 0x7f370408a3c0 con 0x7f37100b1740 2024-03-18T14:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.525+0000 7f370f7fe640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f36fc06bbd0 0x7f36fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T14:44:45.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.525+0000 7f370f7fe640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f36fc06bbd0 0x7f36fc06e090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f37100cbe40 tx=0x7f3700008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T14:44:45.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.529+0000 7f36f7fff640 1 -- 172.21.15.6:0/125286451 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3704059020 con 0x7f37100b1740 2024-03-18T14:44:45.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.751+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f371006bfa0 con 0x7f37100b1740 2024-03-18T14:44:45.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.752+0000 7f36f7fff640 1 -- 172.21.15.6:0/125286451 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3857 (secure 0 0 0) 0x7f3704058920 con 0x7f37100b1740 2024-03-18T14:44:45.753 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-18T14:44:45.753 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5528,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/1499402578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1499402578},{"type":"v1","addr":"172.21.15.92:6836","nonce":1499402578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6022,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/300544617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":300544617},{"type":"v1","addr":"172.21.15.6:6835","nonce":300544617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6034,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/175151750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":175151750},{"type":"v1","addr":"172.21.15.92:6837","nonce":175151750}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6687,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3475300774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3475300774},{"type":"v1","addr":"172.21.15.6:6834","nonce":3475300774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-03-18T14:44:45.754 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-18T14:44:45.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f36fc06bbd0 msgr2=0x7f36fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f36fc06bbd0 0x7f36fc06e090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f37100cbe40 tx=0x7f3700008040 comp rx=0 tx=0).stop 2024-03-18T14:44:45.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37100b1740 msgr2=0x7f37101e9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T14:44:45.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37100b1740 0x7f37101e9580 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f370400b4a0 tx=0x7f370400b960 comp rx=0 tx=0).stop 2024-03-18T14:44:45.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 shutdown_connections 2024-03-18T14:44:45.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:6832/4016547990,v1:172.21.15.92:6833/4016547990] conn(0x7f36fc06bbd0 0x7f36fc06e090 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37100b1b20 0x7f37101dc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37100b1740 0x7f37101e9580 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 --2- 172.21.15.6:0/125286451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371006b790 0x7f37100b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T14:44:45.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 >> 172.21.15.6:0/125286451 conn(0x7f37100d09a0 msgr2=0x7f37101e5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T14:44:45.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 shutdown_connections 2024-03-18T14:44:45.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-18T14:44:45.754+0000 7f3715ccc640 1 -- 172.21.15.6:0/125286451 wait complete. 2024-03-18T14:44:45.766 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-18T14:44:45.766 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-18T14:44:45.766 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T14:44:45.766 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:45.775 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T14:44:45.776 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T14:44:45.776 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:45.826 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T14:44:45.826 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T14:44:45.826 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:45.837 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T14:44:45.838 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T14:44:45.838 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:45.889 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T14:44:45.890 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-18T14:44:45.890 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-18T14:44:45.890 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:45.922 INFO:tasks.ceph.osd.0:Stopped 2024-03-18T14:44:45.923 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-18T14:44:45.923 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:45.992 INFO:tasks.ceph.osd.1:Stopped 2024-03-18T14:44:45.992 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-18T14:44:45.992 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.059 INFO:tasks.ceph.osd.2:Stopped 2024-03-18T14:44:46.059 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-18T14:44:46.059 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.126 INFO:tasks.ceph.osd.3:Stopped 2024-03-18T14:44:46.126 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-18T14:44:46.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.160 INFO:tasks.ceph.osd.4:Stopped 2024-03-18T14:44:46.161 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-18T14:44:46.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.235 INFO:tasks.ceph.osd.5:Stopped 2024-03-18T14:44:46.235 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-18T14:44:46.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.305 INFO:tasks.ceph.osd.6:Stopped 2024-03-18T14:44:46.305 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-18T14:44:46.305 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.382 INFO:tasks.ceph.osd.7:Stopped 2024-03-18T14:44:46.382 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-18T14:44:46.383 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-18T14:44:46.383 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.410 INFO:tasks.ceph.mgr.y:Stopped 2024-03-18T14:44:46.410 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-18T14:44:46.410 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.453 INFO:tasks.ceph.mgr.x:Stopped 2024-03-18T14:44:46.453 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-18T14:44:46.453 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-18T14:44:46.453 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.465 INFO:tasks.ceph.mon.a:Stopped 2024-03-18T14:44:46.465 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-18T14:44:46.465 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.505 INFO:tasks.ceph.mon.b:Stopped 2024-03-18T14:44:46.506 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-18T14:44:46.506 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T14:44:46.558 INFO:tasks.ceph.mon.c:Stopped 2024-03-18T14:44:46.558 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-18T14:44:46.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T14:44:46.589 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:46.590 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-18T14:44:46.763 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:46.763 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-18T14:44:46.846 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:46.846 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-18T14:44:46.895 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:46.895 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-18T14:44:46.945 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:44:46.946 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-18T14:44:47.420 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:44:47.420 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-18T14:44:47.517 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:44:47.518 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-18T14:44:47.574 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:44:47.575 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-18T14:44:47.633 INFO:tasks.ceph:Archiving mon data... 2024-03-18T14:44:47.634 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/data/mon.a.tgz 2024-03-18T14:44:47.634 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-18T14:44:47.650 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.yHE3H958Iu 2024-03-18T14:44:47.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.yHE3H958Iu 2024-03-18T14:44:47.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.yHE3H958Iu 2024-03-18T14:44:47.912 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.yHE3H958Iu is 357KB 2024-03-18T14:44:48.009 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.yHE3H958Iu 2024-03-18T14:44:48.023 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/data/mon.b.tgz 2024-03-18T14:44:48.023 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-18T14:44:48.039 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.MmVbLCpgqo 2024-03-18T14:44:48.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.MmVbLCpgqo 2024-03-18T14:44:48.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.MmVbLCpgqo 2024-03-18T14:44:48.289 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.MmVbLCpgqo is 470KB 2024-03-18T14:44:48.385 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.MmVbLCpgqo 2024-03-18T14:44:48.400 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/data/mon.c.tgz 2024-03-18T14:44:48.400 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-18T14:44:48.454 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.PuRhf8L1il 2024-03-18T14:44:48.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.PuRhf8L1il 2024-03-18T14:44:48.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.PuRhf8L1il 2024-03-18T14:44:48.694 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.PuRhf8L1il is 470KB 2024-03-18T14:44:48.762 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.PuRhf8L1il 2024-03-18T14:44:48.776 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-18T14:44:48.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-18T14:44:48.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-18T14:44:48.864 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-18T14:44:48.898 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-18T14:44:48.898 INFO:tasks.ceph:Archiving crash dumps... 2024-03-18T14:44:48.900 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/remote/smithi006/crash 2024-03-18T14:44:48.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-18T14:44:48.933 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/remote/smithi092/crash 2024-03-18T14:44:48.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-18T14:44:48.964 INFO:tasks.ceph:Shutting down logrotate 2024-03-18T14:44:48.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:48.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:49.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:49.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-18T14:44:49.108 INFO:tasks.ceph:Compressing logs... 2024-03-18T14:44:49.108 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T14:44:49.124 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T14:44:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58064.log 2024-03-18T14:44:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49342.log 2024-03-18T14:44:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48356.log 2024-03-18T14:44:49.151 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58064.log.gz 2024-03-18T14:44:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55307.log 2024-03-18T14:44:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48356.log: gzip -5 88.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49342.log.gz /var/log/ceph/ceph-client.admin.55391.log 2024-03-18T14:44:49.152 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.198 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-18T14:44:49.199 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44164.log 2024-03-18T14:44:49.199 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37676.log 2024-03-18T14:44:49.199 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.1.44164.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-18T14:44:49.199 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37676.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-18T14:44:49.199 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-18T14:44:49.199 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.audit.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37676.log.gz 2024-03-18T14:44:49.199 INFO:teuthology.orchestra.run.smithi092.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.1.44164.log.gz 2024-03-18T14:44:49.200 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44621.log 2024-03-18T14:44:49.200 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-18T14:44:49.200 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.3.44621.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-18T14:44:49.200 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37516.log 2024-03-18T14:44:49.200 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37596.log 2024-03-18T14:44:49.200 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: 91.1% 91.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr: -- replaced with /var/log/ceph/ceph-client.3.44621.log.gz/var/log/ceph/ceph-client.admin.37516.log: 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37516.log.gz 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37396.log 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37596.log.gz 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-18T14:44:49.201 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5/var/log/ceph/ceph-client.admin.37396.log: --verbose -- /var/log/ceph/ceph-client.admin.37476.log 2024-03-18T14:44:49.202 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37396.log.gz 2024-03-18T14:44:49.202 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-18T14:44:49.202 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37436.log 2024-03-18T14:44:49.202 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.37476.log: 92.0% 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37476.log.gz 2024-03-18T14:44:49.202 INFO:teuthology.orchestra.run.smithi092.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-18T14:44:49.202 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-18T14:44:49.202 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37556.log 2024-03-18T14:44:49.203 INFO:teuthology.orchestra.run.smithi092.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37436.log.gz 2024-03-18T14:44:49.203 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2024-03-18T14:44:49.203 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37556.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37556.log.gz 2024-03-18T14:44:49.203 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-18T14:44:49.203 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37636.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz 2024-03-18T14:44:49.220 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.c.log: 92.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-18T14:44:49.246 INFO:teuthology.orchestra.run.smithi092.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-18T14:44:49.286 INFO:teuthology.orchestra.run.smithi092.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-18T14:44:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57043.log 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48356.log.gz 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5%/var/log/ceph/ceph-client.admin.55391.log: -- replaced with /var/log/ceph/ceph-client.admin.55307.log.gz 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60623.log 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57043.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55391.log.gz 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45891.log 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60623.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57043.log.gz 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56794.log 2024-03-18T14:44:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45891.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60623.log.gz 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45891.log.gz 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.56794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58106.log 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50887.log 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49552.log 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gz 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58106.log.gz 2024-03-18T14:44:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50887.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-18T14:44:49.310 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50887.log.gz 2024-03-18T14:44:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49552.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.56794.log.gz 2024-03-18T14:44:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50244.log 2024-03-18T14:44:49.310 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3%/var/log/ceph/ceph-mon.a.log: -- replaced with /var/log/ceph/ceph-client.admin.49552.log.gz 2024-03-18T14:44:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45297.log 2024-03-18T14:44:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57557.log 2024-03-18T14:44:49.311 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50244.log.gz 2024-03-18T14:44:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51381.log 2024-03-18T14:44:49.311 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45297.log.gz 2024-03-18T14:44:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45213.log 2024-03-18T14:44:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57557.log: /var/log/ceph/ceph-client.admin.51381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57311.log 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57557.log.gz 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51381.log.gz 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47843.log 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45213.log.gz 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53768.log 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47843.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.57311.log.gz -5 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53810.log 2024-03-18T14:44:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53768.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47843.log.gz 2024-03-18T14:44:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49174.log 2024-03-18T14:44:49.313 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53768.log.gz 2024-03-18T14:44:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43983.log 2024-03-18T14:44:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49174.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.53810.log.gz -5 2024-03-18T14:44:49.313 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56246.log 2024-03-18T14:44:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43983.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49174.log.gz 2024-03-18T14:44:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52417.log 2024-03-18T14:44:49.314 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43983.log.gz 2024-03-18T14:44:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55349.log 2024-03-18T14:44:49.314 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56246.log.gz 2024-03-18T14:44:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59118.log 2024-03-18T14:44:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55349.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52417.log.gz 2024-03-18T14:44:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45585.log 2024-03-18T14:44:49.314 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55349.log.gz 2024-03-18T14:44:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58208.log 2024-03-18T14:44:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45933.log 2024-03-18T14:44:49.315 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59118.log.gz 2024-03-18T14:44:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58208.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45585.log.gz 2024-03-18T14:44:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63672.log 2024-03-18T14:44:49.315 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58208.log.gz 2024-03-18T14:44:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46109.log 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63672.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45933.log.gz 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57980.log 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63672.log.gz 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57896.log 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46109.log.gz 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52711.log 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57896.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57980.log.gz 2024-03-18T14:44:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55978.log 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57896.log.gz 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57854.log 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52711.log.gz 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46781.log 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57854.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55978.log.gz 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58545.log 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57854.log.gz 2024-03-18T14:44:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58419.log 2024-03-18T14:44:49.318 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46781.log.gz 2024-03-18T14:44:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51913.log 2024-03-18T14:44:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58419.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58545.log.gz 2024-03-18T14:44:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63587.log 2024-03-18T14:44:49.318 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58419.log.gz 2024-03-18T14:44:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53726.log 2024-03-18T14:44:49.318 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51913.log.gz 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57437.log 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53726.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63587.log.gz 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58335.log 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53726.log.gz 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61340.log 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57437.log.gz 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50929.log 2024-03-18T14:44:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61340.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58335.log.gz 2024-03-18T14:44:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46435.log 2024-03-18T14:44:49.320 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61340.log.gz 2024-03-18T14:44:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62437.log 2024-03-18T14:44:49.320 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50929.log.gz 2024-03-18T14:44:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55081.log 2024-03-18T14:44:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47193.log 2024-03-18T14:44:49.320 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46435.log.gz 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62437.log.gz 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59659.log 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55081.log.gz 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61056.log 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59659.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47193.log.gz 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61114.log 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9%/var/log/ceph/ceph-client.admin.61056.log: -- replaced with /var/log/ceph/ceph-client.admin.59659.log.gz 2024-03-18T14:44:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56104.log 2024-03-18T14:44:49.322 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61056.log.gz 2024-03-18T14:44:49.322 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50473.log 2024-03-18T14:44:49.322 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56104.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61114.log.gz 2024-03-18T14:44:49.322 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51871.log 2024-03-18T14:44:49.322 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56104.log.gz 2024-03-18T14:44:49.322 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46111.log 2024-03-18T14:44:49.322 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50473.log.gz 2024-03-18T14:44:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59593.log 2024-03-18T14:44:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46111.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51871.log.gz 2024-03-18T14:44:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46786.log 2024-03-18T14:44:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59593.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46111.log.gz 2024-03-18T14:44:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52963.log 2024-03-18T14:44:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43581.log 2024-03-18T14:44:49.323 INFO:teuthology.orchestra.run.smithi006.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59593.log.gz 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52963.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46786.log.gz 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60545.log 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52963.log.gz 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59341.log 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43581.log: /var/log/ceph/ceph-client.admin.60545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47379.log 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43581.log.gz 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60545.log.gz 2024-03-18T14:44:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51793.log 2024-03-18T14:44:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47379.log: gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.59341.log.gz 2024-03-18T14:44:49.325 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.0.54476.log 2024-03-18T14:44:49.325 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47379.log.gz 2024-03-18T14:44:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47631.log 2024-03-18T14:44:49.325 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51793.log.gz 2024-03-18T14:44:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.54476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49888.log 2024-03-18T14:44:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47505.log 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49888.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47631.log.gz 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57686.log 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58903.log 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49888.log.gz 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47505.log: /var/log/ceph/ceph-client.admin.57686.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47505.log.gz 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52249.log 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57686.log.gz 2024-03-18T14:44:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55039.log 2024-03-18T14:44:49.327 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58903.log: /var/log/ceph/ceph-client.admin.52249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56146.log 2024-03-18T14:44:49.327 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58903.log.gz 2024-03-18T14:44:49.327 INFO:teuthology.orchestra.run.smithi006.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.54476.log.gz 2024-03-18T14:44:49.327 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55039.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52249.log.gz 2024-03-18T14:44:49.327 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63240.log 2024-03-18T14:44:49.327 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55039.log.gz 2024-03-18T14:44:49.327 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58022.log 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56146.log.gz 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52585.log 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63240.log.gz 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52501.log 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58022.log: /var/log/ceph/ceph-client.admin.52585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60883.log 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58022.log.gz 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52501.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52585.log.gz 2024-03-18T14:44:49.328 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44025.log 2024-03-18T14:44:49.329 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52501.log.gz 2024-03-18T14:44:49.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.60883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59299.log 2024-03-18T14:44:49.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49426.log 2024-03-18T14:44:49.329 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44025.log.gz 2024-03-18T14:44:49.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61198.log 2024-03-18T14:44:49.329 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59299.log.gz 2024-03-18T14:44:49.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46784.log 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.60883.log.gz 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49426.log.gz 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59425.log 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61198.log.gz 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46446.log 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59425.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46784.log.gz 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58988.log 2024-03-18T14:44:49.330 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59425.log.gz 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51139.log 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46446.log.gz 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60319.log 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51139.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58988.log.gz 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50845.log 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51139.log.gz 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53085.log 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60319.log.gz 2024-03-18T14:44:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53293.log 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53085.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50845.log.gz 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57269.log 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53085.log.gz 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53293.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31265.log 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53293.log.gz 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52669.log 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57269.log: /var/log/ceph/ceph.tmp-client.admin.31265.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31265.log.gz 2024-03-18T14:44:49.332 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62395.log 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57269.log.gz 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43319.log 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52669.log.gz 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58780.log 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62395.log.gz 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58780.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43319.log.gz 2024-03-18T14:44:49.333 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46106.log 2024-03-18T14:44:49.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43665.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58780.log.gz 2024-03-18T14:44:49.334 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50056.log 2024-03-18T14:44:49.334 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2024-03-18T14:44:49.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61424.log 2024-03-18T14:44:49.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59169.log 2024-03-18T14:44:49.334 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46106.log.gz 2024-03-18T14:44:49.334 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50056.log.gz 2024-03-18T14:44:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56372.log 2024-03-18T14:44:49.335 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61424.log.gz 2024-03-18T14:44:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63966.log 2024-03-18T14:44:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56372.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59169.log.gz 2024-03-18T14:44:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37295.log 2024-03-18T14:44:49.335 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5%/var/log/ceph/ceph-client.admin.63966.log: -- replaced with /var/log/ceph/ceph-client.admin.56372.log.gz 2024-03-18T14:44:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50683.log 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63966.log.gz 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62875.log 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50683.log: gzip 89.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.37295.log.gz -- 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.52543.log 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50683.log.gz 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56288.log 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62875.log.gz 2024-03-18T14:44:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56062.log 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56288.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52543.log.gz 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46451.log 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56288.log.gz 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62918.log 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54216.log 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56062.log.gz 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62918.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46451.log.gz 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47715.log 2024-03-18T14:44:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54216.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62918.log.gz 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43897.log 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54216.log.gz 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46115.log 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59509.log 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46115.log: 88.4% 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43897.log.gz -- replaced with /var/log/ceph/ceph-client.admin.47715.log.gz 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46113.log 2024-03-18T14:44:49.338 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46115.log.gz 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48732.log 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59509.log.gz 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63044.log 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.2.48732.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46113.log.gz 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47337.log 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46108.log 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63044.log.gz 2024-03-18T14:44:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47337.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.2.48732.log.gz 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47337.log.gz 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52753.log 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46108.log.gz 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47589.log 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50373.log 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47589.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52753.log.gz 2024-03-18T14:44:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63756.log 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47589.log.gz 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47235.log 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50373.log: /var/log/ceph/ceph-client.admin.63756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62731.log 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50373.log.gz 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47235.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63756.log.gz 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37545.log 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58945.log 2024-03-18T14:44:49.341 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47235.log.gz 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62731.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37545.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.62731.log.gz 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49720.log 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8%/var/log/ceph/ceph-client.admin.58945.log: -- replaced with /var/log/ceph/ceph-client.admin.37545.log.gz 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45723.log 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58945.log.gz 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.49720.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64008.log 2024-03-18T14:44:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45723.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49720.log.gz 2024-03-18T14:44:49.343 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50761.log 2024-03-18T14:44:49.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.64008.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45723.log.gz 2024-03-18T14:44:49.343 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62038.log 2024-03-18T14:44:49.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50761.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64008.log.gz 2024-03-18T14:44:49.343 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62285.log 2024-03-18T14:44:49.343 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50761.log.gz 2024-03-18T14:44:49.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47421.log 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51339.log 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62038.log.gz 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62285.log.gz 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63840.log 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47421.log.gz 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57479.log 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63840.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51339.log.gz 2024-03-18T14:44:49.344 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51667.log 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57479.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63840.log.gz 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63002.log 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57479.log.gz 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51709.log 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63002.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.51667.log.gz 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53503.log 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51709.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63002.log.gz 2024-03-18T14:44:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55123.log 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51709.log.gz 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53898.log 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53503.log.gz 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64110.log 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53898.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.55123.log.gz --verbose 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.51097.log 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53898.log.gz 2024-03-18T14:44:49.346 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.64110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49930.log 2024-03-18T14:44:49.347 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64110.log.gz 2024-03-18T14:44:49.347 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57211.log 2024-03-18T14:44:49.347 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45636.log 2024-03-18T14:44:49.347 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51097.log.gz 2024-03-18T14:44:49.347 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49930.log.gz 2024-03-18T14:44:49.347 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58293.log 2024-03-18T14:44:49.347 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57211.log.gz 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54108.log 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58293.log: gzip -5 88.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45636.log.gz /var/log/ceph/ceph-client.0.48234.log 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58293.log.gz 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47109.log 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54108.log.gz 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.48234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63545.log 2024-03-18T14:44:49.348 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59253.log 2024-03-18T14:44:49.349 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47109.log.gz 2024-03-18T14:44:49.349 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46779.log 2024-03-18T14:44:49.349 INFO:teuthology.orchestra.run.smithi006.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.0.48234.log.gz 2024-03-18T14:44:49.349 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63545.log.gz 2024-03-18T14:44:49.349 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50803.log 2024-03-18T14:44:49.349 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59253.log: /var/log/ceph/ceph-client.admin.46779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54855.log 2024-03-18T14:44:49.349 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59253.log.gz 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0%/var/log/ceph/ceph-client.admin.50803.log: -- replaced with /var/log/ceph/ceph-client.admin.46779.log.gz 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64050.log 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54855.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50803.log.gz 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59919.log 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.64050.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54855.log.gz 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44277.log 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64050.log.gz 2024-03-18T14:44:49.350 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.59919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60377.log 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52459.log 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46647.log 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60377.log.gz 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44343.log 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.44277.log.gz 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46647.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52459.log.gz 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44945.log 2024-03-18T14:44:49.351 INFO:teuthology.orchestra.run.smithi006.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.59919.log.gz 2024-03-18T14:44:49.352 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63924.log 2024-03-18T14:44:49.352 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46647.log.gz 2024-03-18T14:44:49.352 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44343.log: /var/log/ceph/ceph-client.admin.44945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45543.log 2024-03-18T14:44:49.352 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44343.log.gz 2024-03-18T14:44:49.352 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44945.log.gz 2024-03-18T14:44:49.352 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63924.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-18T14:44:49.352 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45543.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63924.log.gz 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56414.log 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45543.log.gz 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60461.log 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45849.log 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56414.log.gz 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46782.log 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60419.log 2024-03-18T14:44:49.353 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60461.log.gz 2024-03-18T14:44:49.354 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45849.log: /var/log/ceph/ceph-client.admin.46782.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45849.log.gz 2024-03-18T14:44:49.354 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49468.log 2024-03-18T14:44:49.354 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46782.log.gz 2024-03-18T14:44:49.354 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56492.log 2024-03-18T14:44:49.354 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60419.log: gzip -5/var/log/ceph/ceph-client.admin.49468.log: --verbose -- /var/log/ceph/ceph-client.admin.46787.log 2024-03-18T14:44:49.354 INFO:teuthology.orchestra.run.smithi006.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60419.log.gz 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56492.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49468.log.gz 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43707.log 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53587.log 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46787.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56492.log.gz 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43707.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46787.log.gz 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59383.log 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43707.log.gz 2024-03-18T14:44:49.355 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61508.log 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53587.log: /var/log/ceph/ceph-client.admin.59383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45501.log 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53587.log.gz 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59383.log.gz 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54633.log 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49972.log 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61508.log.gz 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45501.log: /var/log/ceph/ceph-client.admin.54633.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45501.log.gz 2024-03-18T14:44:49.356 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49216.log 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62605.log 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49972.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54633.log.gz 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45029.log 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49972.log.gz 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49216.log.gz 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62960.log 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62773.log 2024-03-18T14:44:49.357 INFO:teuthology.orchestra.run.smithi006.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62605.log.gz 2024-03-18T14:44:49.358 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45029.log.gz 2024-03-18T14:44:49.358 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54913.log 2024-03-18T14:44:49.358 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62960.log.gz 2024-03-18T14:44:49.358 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57169.log 2024-03-18T14:44:49.358 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62773.log: /var/log/ceph/ceph-client.admin.54913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51997.log 2024-03-18T14:44:49.358 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62773.log.gz 2024-03-18T14:44:49.358 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57169.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54913.log.gz 2024-03-18T14:44:49.359 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52921.log 2024-03-18T14:44:49.359 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose 88.4% -- /var/log/ceph/ceph-client.admin.43749.log -- replaced with /var/log/ceph/ceph-client.admin.57169.log.gz 2024-03-18T14:44:49.359 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.359 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51997.log: /var/log/ceph/ceph-client.admin.52921.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51997.log.gz 2024-03-18T14:44:49.359 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60503.log 2024-03-18T14:44:49.359 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52921.log.gz 2024-03-18T14:44:49.359 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59467.log 2024-03-18T14:44:49.360 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43749.log: /var/log/ceph/ceph-client.admin.60503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56330.log 2024-03-18T14:44:49.360 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43749.log.gz 2024-03-18T14:44:49.360 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60503.log.gz 2024-03-18T14:44:49.360 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57395.log 2024-03-18T14:44:49.360 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51223.log 2024-03-18T14:44:49.360 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56330.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59467.log.gz 2024-03-18T14:44:49.360 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56020.log 2024-03-18T14:44:49.361 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56330.log.gz 2024-03-18T14:44:49.361 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57395.log.gz 2024-03-18T14:44:49.361 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63501.log 2024-03-18T14:44:49.361 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51223.log.gz 2024-03-18T14:44:49.361 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63882.log 2024-03-18T14:44:49.361 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56020.log: /var/log/ceph/ceph-client.admin.63501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59074.log 2024-03-18T14:44:49.361 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56020.log.gz 2024-03-18T14:44:49.362 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63501.log.gz 2024-03-18T14:44:49.362 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49258.log 2024-03-18T14:44:49.362 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47025.log 2024-03-18T14:44:49.362 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59074.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63882.log.gz 2024-03-18T14:44:49.362 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47463.log 2024-03-18T14:44:49.362 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59074.log.gz 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49258.log.gz 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58148.log 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51583.log 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47025.log.gz 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58148.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47463.log.gz 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61466.log 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58148.log.gz 2024-03-18T14:44:49.363 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52837.log 2024-03-18T14:44:49.364 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51583.log: /var/log/ceph/ceph-client.admin.61466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54779.log 2024-03-18T14:44:49.364 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51583.log.gz 2024-03-18T14:44:49.364 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52837.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61466.log.gz 2024-03-18T14:44:49.364 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60193.log 2024-03-18T14:44:49.364 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52837.log.gz 2024-03-18T14:44:49.364 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52039.log 2024-03-18T14:44:49.364 INFO:teuthology.orchestra.run.smithi006.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54779.log.gz 2024-03-18T14:44:49.365 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63630.log 2024-03-18T14:44:49.365 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60193.log: /var/log/ceph/ceph-client.admin.52039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54066.log 2024-03-18T14:44:49.365 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60193.log.gz 2024-03-18T14:44:49.365 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63630.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52039.log.gz 2024-03-18T14:44:49.365 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51055.log 2024-03-18T14:44:49.365 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63630.log.gz 2024-03-18T14:44:49.366 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45765.log 2024-03-18T14:44:49.366 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54066.log: /var/log/ceph/ceph-client.admin.51055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57127.log 2024-03-18T14:44:49.366 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54066.log.gz 2024-03-18T14:44:49.366 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45765.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51055.log.gz 2024-03-18T14:44:49.366 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59551.log 2024-03-18T14:44:49.366 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45765.log.gz 2024-03-18T14:44:49.366 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55469.log 2024-03-18T14:44:49.367 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57127.log: /var/log/ceph/ceph-client.admin.59551.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-18T14:44:49.367 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57127.log.gz 2024-03-18T14:44:49.367 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55469.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59551.log.gz 2024-03-18T14:44:49.367 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49594.log 2024-03-18T14:44:49.367 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55469.log.gz 2024-03-18T14:44:49.367 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51625.log 2024-03-18T14:44:49.367 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.49594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46445.log 2024-03-18T14:44:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51625.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49594.log.gz 2024-03-18T14:44:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53631.log 2024-03-18T14:44:49.368 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51625.log.gz 2024-03-18T14:44:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63324.log 2024-03-18T14:44:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46445.log: /var/log/ceph/ceph-client.admin.53631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52165.log 2024-03-18T14:44:49.368 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46445.log.gz 2024-03-18T14:44:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63324.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53631.log.gz 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52627.log 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63324.log.gz 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57728.log 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52165.log.gz 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48896.log 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52627.log: /var/log/ceph/ceph-client.admin.57728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44151.log 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52627.log.gz 2024-03-18T14:44:49.369 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48896.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57728.log.gz 2024-03-18T14:44:49.370 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45681.log 2024-03-18T14:44:49.370 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48896.log.gz 2024-03-18T14:44:49.370 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63409.log 2024-03-18T14:44:49.370 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57938.log 2024-03-18T14:44:49.370 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2%/var/log/ceph/ceph-client.admin.45681.log: -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz 2024-03-18T14:44:49.370 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63409.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45681.log.gz 2024-03-18T14:44:49.370 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47673.log 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63409.log.gz 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44067.log 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57938.log: /var/log/ceph/ceph-client.admin.47673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55265.log 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57938.log.gz 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47673.log.gz 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54024.log 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54150.log 2024-03-18T14:44:49.371 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55265.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44067.log.gz 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54024.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55265.log.gz 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50971.log 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43496.log 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54150.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54024.log.gz 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63459.log 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50971.log.gz 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63198.log 2024-03-18T14:44:49.372 INFO:teuthology.orchestra.run.smithi006.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54150.log.gz 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63459.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43496.log.gz 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44235.log 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45807.log 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63459.log.gz 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63198.log.gz 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49006.log 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44235.log.gz 2024-03-18T14:44:49.373 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62117.log 2024-03-18T14:44:49.374 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45807.log: /var/log/ceph/ceph-client.admin.49006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53166.log 2024-03-18T14:44:49.374 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45807.log.gz 2024-03-18T14:44:49.374 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62117.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49006.log.gz 2024-03-18T14:44:49.374 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63798.log 2024-03-18T14:44:49.374 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62117.log.gz 2024-03-18T14:44:49.374 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54677.log 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53166.log.gz 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49678.log 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63798.log: /var/log/ceph/ceph-client.admin.54677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60277.log 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63798.log.gz 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49678.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43415.log 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54677.log.gz 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49678.log.gz 2024-03-18T14:44:49.375 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51297.log 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60277.log.gz 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49090.log 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43415.log: /var/log/ceph/ceph-client.admin.51297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62159.log 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43415.log.gz 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51297.log.gz 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47277.log 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57085.log 2024-03-18T14:44:49.376 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49090.log.gz 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47277.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62159.log.gz 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51483.log 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47277.log.gz 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53982.log 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57085.log: /var/log/ceph/ceph-client.admin.51483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51423.log 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57085.log.gz 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51483.log.gz 2024-03-18T14:44:49.377 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58377.log 2024-03-18T14:44:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47067.log 2024-03-18T14:44:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51423.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53982.log.gz 2024-03-18T14:44:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58377.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51423.log.gz 2024-03-18T14:44:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53005.log 2024-03-18T14:44:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44987.log 2024-03-18T14:44:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47067.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58377.log.gz 2024-03-18T14:44:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53005.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47067.log.gz 2024-03-18T14:44:49.379 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47151.log 2024-03-18T14:44:49.379 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54997.log 2024-03-18T14:44:49.379 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44987.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53005.log.gz 2024-03-18T14:44:49.379 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61282.log 2024-03-18T14:44:49.379 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44987.log.gz 2024-03-18T14:44:49.379 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47151.log.gz 2024-03-18T14:44:49.379 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52333.log 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54997.log: /var/log/ceph/ceph-client.admin.61282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50557.log 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54997.log.gz 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52333.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61282.log.gzgzip 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53419.log 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52333.log.gz 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50557.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50557.log.gz 2024-03-18T14:44:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53335.log 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53419.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61864.log 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53419.log.gz 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50194.log 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.61864.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53335.log.gz 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58461.log 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62563.log 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7%/var/log/ceph/ceph-client.admin.58461.log: -- replaced with /var/log/ceph/ceph-client.admin.50194.log.gz 2024-03-18T14:44:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44753.log 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58461.log.gz 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49132.log 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.61864.log.gz 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50641.log 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44753.log: /var/log/ceph/ceph-client.admin.49132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49048.log 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62563.log.gz 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50641.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49132.log.gz 2024-03-18T14:44:49.382 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52207.log 2024-03-18T14:44:49.383 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50641.log.gz 2024-03-18T14:44:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57001.log 2024-03-18T14:44:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49048.log: /var/log/ceph/ceph-client.admin.52207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64192.log 2024-03-18T14:44:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57001.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49048.log.gz 2024-03-18T14:44:49.383 INFO:teuthology.orchestra.run.smithi006.stderr: 92.6% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52207.log.gz 2024-03-18T14:44:49.383 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44753.log.gz 2024-03-18T14:44:49.383 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57001.log.gz 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54955.log 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.64192.log: /var/log/ceph/ceph-client.admin.63282.log 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54955.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64192.log.gz 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50098.log 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54955.log.gz 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45071.log 2024-03-18T14:44:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63282.log: /var/log/ceph/ceph-client.admin.50098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43538.log 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63282.log.gz 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50098.log.gz 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57353.log 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43793.log 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43538.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45071.log.gz 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57353.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43538.log.gz 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55223.log 2024-03-18T14:44:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43793.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57353.log.gz 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51955.log 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63714.log 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43793.log.gz 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55223.log.gz 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43623.log 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43837.log 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63714.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51955.log.gz 2024-03-18T14:44:49.386 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64150.log 88.5% 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63714.log.gz 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43837.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.50515.log 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43623.log.gz 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.64150.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43837.log.gz 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45423.log 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64150.log.gz 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61604.log 2024-03-18T14:44:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55729.log 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50515.log.gz 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61604.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45423.log.gz 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61604.log.gz 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.55729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62689.log 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43939.log 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.62689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52879.log 2024-03-18T14:44:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43939.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62689.log.gz 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60235.log 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr: 91.0% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43939.log.gz 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.0.55729.log.gz 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58605.log 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52879.log.gz 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50330.log 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60235.log: /var/log/ceph/ceph-client.admin.58605.log: gzip -5 --verbose -- 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60235.log.gz 2024-03-18T14:44:49.389 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.58861.log 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5%/var/log/ceph/ceph-client.admin.50330.log: -- replaced with /var/log/ceph/ceph-client.admin.58605.log.gz 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53545.log 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50330.log.gz 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44109.log 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.53545.log: --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58861.log.gz 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.46447.log 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44109.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53545.log.gz 2024-03-18T14:44:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44109.log.gz 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.53675.log 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46447.log: gzip 88.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.50599.log 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46447.log.gz 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50014.log 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53675.log.gz 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62201.log 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.50014.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57770.log 2024-03-18T14:44:49.391 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50599.log.gz 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62201.log: gzip -5 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50014.log.gz 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58503.log 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62201.log.gz 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52375.log 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57770.log.gz 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45155.log 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52375.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58503.log.gz 2024-03-18T14:44:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52081.log 2024-03-18T14:44:49.393 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52375.log.gz 2024-03-18T14:44:49.393 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46450.log 2024-03-18T14:44:49.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45155.log: /var/log/ceph/ceph-client.admin.52081.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45155.log.gz 2024-03-18T14:44:49.393 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52795.log 2024-03-18T14:44:49.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46450.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52081.log.gz 2024-03-18T14:44:49.393 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-18T14:44:49.393 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46450.log.gz 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62647.log 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.52795.log.gz 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51013.log 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59211.log 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51013.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62647.log.gz 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr: 85.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53208.log 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-18T14:44:49.394 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51013.log.gz 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59211.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59211.log.gz 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.53208.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.53208.log.gz 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62479.log 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46785.log 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62479.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2024-03-18T14:44:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-18T14:44:49.396 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62479.log.gz 2024-03-18T14:44:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49510.log 2024-03-18T14:44:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46785.log: /var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53377.log 2024-03-18T14:44:49.396 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46785.log.gz/var/log/ceph/ceph-client.admin.49510.log: 2024-03-18T14:44:49.396 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54719.log 2024-03-18T14:44:49.396 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49510.log.gz 2024-03-18T14:44:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55936.log 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54719.log: 89.4% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54719.log.gz 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53377.log.gz 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58251.log 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46983.log 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55936.log.gz 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53250.log 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58251.log.gz 2024-03-18T14:44:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46448.log 2024-03-18T14:44:49.398 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46983.log.gz 2024-03-18T14:44:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53250.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53250.log.gz 2024-03-18T14:44:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46107.log 2024-03-18T14:44:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45381.log 2024-03-18T14:44:49.398 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46448.log.gz 2024-03-18T14:44:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46107.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46107.log.gz 2024-03-18T14:44:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56534.log 2024-03-18T14:44:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47547.log 2024-03-18T14:44:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45381.log: /var/log/ceph/ceph-client.admin.56534.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45381.log.gz 2024-03-18T14:44:49.399 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61240.log 2024-03-18T14:44:49.399 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56534.log.gz 2024-03-18T14:44:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47547.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47547.log.gz 2024-03-18T14:44:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46443.log 2024-03-18T14:44:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49384.log 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61240.log: /var/log/ceph/ceph-client.admin.46443.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61240.log.gz 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48854.log 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46443.log.gz 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49384.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49384.log.gz 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44193.log 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53940.log 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48854.log: /var/log/ceph/ceph-client.admin.44193.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48854.log.gz 2024-03-18T14:44:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49804.log 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44193.log.gz 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53940.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53940.log.gz 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51525.log 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50288.log 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49804.log: /var/log/ceph/ceph-client.admin.51525.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49804.log.gz 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51525.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51751.log 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50288.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50288.log.gz 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49300.log 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53856.log 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51751.log: /var/log/ceph/ceph-client.admin.49300.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51751.log.gz 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50415.log 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49300.log.gz 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53856.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53856.log.gz 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43238.log 2024-03-18T14:44:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45339.log 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50415.log: /var/log/ceph/ceph-client.admin.43238.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50415.log.gz 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49636.log 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43238.log.gz 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45339.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45339.log.gz 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53461.log 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52123.log 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49636.log: /var/log/ceph/ceph-client.admin.53461.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49636.log.gz 2024-03-18T14:44:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:gzip 89.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.63367.log 2024-03-18T14:44:49.404 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53461.log.gz 2024-03-18T14:44:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52123.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52123.log.gz 2024-03-18T14:44:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57812.log 2024-03-18T14:44:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49846.log 2024-03-18T14:44:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.63367.log: /var/log/ceph/ceph-client.admin.57812.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63367.log.gz 2024-03-18T14:44:49.404 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57812.log.gz 2024-03-18T14:44:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55181.log 2024-03-18T14:44:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49846.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49846.log.gz 2024-03-18T14:44:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51181.log 2024-03-18T14:44:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60151.log 2024-03-18T14:44:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55181.log: /var/log/ceph/ceph-client.admin.51181.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55181.log.gz 2024-03-18T14:44:49.405 INFO:teuthology.orchestra.run.smithi006.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51181.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61156.log 2024-03-18T14:44:49.405 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60151.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60151.log.gz 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56204.log 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49762.log 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61156.log: /var/log/ceph/ceph-client.admin.56204.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61156.log.gz 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56204.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62243.log 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49762.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49762.log.gz 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62521.log 2024-03-18T14:44:49.406 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59030.log 2024-03-18T14:44:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62243.log: /var/log/ceph/ceph-client.admin.62521.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62243.log.gz 2024-03-18T14:44:49.407 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60093.log 2024-03-18T14:44:49.407 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62521.log.gz 2024-03-18T14:44:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59030.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59030.log.gz 2024-03-18T14:44:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-18T14:44:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52291.log 2024-03-18T14:44:49.407 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.60093.log: /var/log/ceph/ceph-mds.a.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60093.log.gz 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62815.log 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52291.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52291.log.gz 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61382.log 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.62815.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62815.log.gz 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46112.log 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45255.log 2024-03-18T14:44:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.61382.log: /var/log/ceph/ceph-client.admin.46112.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61382.log.gz 2024-03-18T14:44:49.409 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46112.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58700.log 2024-03-18T14:44:49.409 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:49.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45255.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45255.log.gz 2024-03-18T14:44:49.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58700.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58700.log.gz 2024-03-18T14:44:49.409 INFO:teuthology.orchestra.run.smithi006.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-18T14:44:49.410 INFO:teuthology.orchestra.run.smithi092.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-18T14:44:49.525 INFO:teuthology.orchestra.run.smithi006.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-18T14:44:49.842 INFO:teuthology.orchestra.run.smithi092.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-18T14:44:49.910 INFO:teuthology.orchestra.run.smithi092.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-18T14:44:49.912 INFO:teuthology.orchestra.run.smithi006.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-18T14:44:49.928 INFO:teuthology.orchestra.run.smithi006.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-18T14:44:49.934 INFO:teuthology.orchestra.run.smithi092.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-18T14:44:49.984 INFO:teuthology.orchestra.run.smithi092.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-18T14:44:49.986 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-03-18T14:44:49.986 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m0.824s 2024-03-18T14:44:49.986 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m3.243s 2024-03-18T14:44:49.986 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m0.138s 2024-03-18T14:44:49.992 INFO:teuthology.orchestra.run.smithi006.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-18T14:44:50.050 INFO:teuthology.orchestra.run.smithi006.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-18T14:44:50.052 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-18T14:44:50.052 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m0.915s 2024-03-18T14:44:50.052 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m3.475s 2024-03-18T14:44:50.052 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.326s 2024-03-18T14:44:50.053 INFO:tasks.ceph:Archiving logs... 2024-03-18T14:44:50.053 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/remote/smithi006/log 2024-03-18T14:44:50.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T14:44:50.877 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/remote/smithi092/log 2024-03-18T14:44:50.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T14:44:51.355 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T14:44:51.364 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T14:44:51.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T14:44:51.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T14:44:51.456 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T14:44:51.465 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T14:44:51.465 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T14:44:51.468 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T14:44:51.481 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-18T14:44:51.483 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-18T14:44:51.487 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T14:44:51.487 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-18T14:44:51.487 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1871us[+1876us] +/- 92ms 2024-03-18T14:44:51.487 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -970us[ -965us] +/- 62ms 2024-03-18T14:44:51.487 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -360us[ -356us] +/- 85ms 2024-03-18T14:44:51.487 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:44:51.488 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T14:44:51.488 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-18T14:44:51.488 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +1855us[+1866us] +/- 92ms 2024-03-18T14:44:51.489 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -969us[ -959us] +/- 62ms 2024-03-18T14:44:51.489 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -355us[ -345us] +/- 85ms 2024-03-18T14:44:51.489 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T14:44:51.489 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T14:44:51.498 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T14:44:51.498 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T14:44:51.508 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T14:44:51.612 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T14:44:51.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:44:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:51.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:44:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:51.747 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T14:44:51.758 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T14:44:51.767 INFO:teuthology.task.internal:Duration was 685.887581 seconds 2024-03-18T14:44:51.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T14:44:51.776 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T14:44:51.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T14:44:51.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T14:44:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:44:51.831 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:44:52.124 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T14:44:52.124 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-03-18T14:44:52.124 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T14:44:52.158 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-03-18T14:44:52.158 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T14:44:52.182 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T14:44:52.183 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T14:44:52.200 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T14:44:52.251 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T14:44:52.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T14:44:52.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T14:44:52.390 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T14:44:52.400 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T14:44:52.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T14:44:52.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T14:44:52.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T14:44:52.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T14:44:52.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T14:44:52.491 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-03-18T14:44:52.519 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-03-18T14:44:52.535 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T14:44:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:52.558 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T14:44:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:44:52.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T14:44:52.601 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T14:44:52.602 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/remote/smithi006 2024-03-18T14:44:52.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T14:44:52.638 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831/remote/smithi092 2024-03-18T14:44:52.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T14:44:52.673 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T14:44:52.673 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T14:44:52.675 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T14:44:52.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T14:44:52.734 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T14:44:52.734 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T14:44:52.744 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T14:44:52.744 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T14:44:52.746 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T14:44:52.758 INFO:teuthology.orchestra.run.smithi006.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 14:44 /home/ubuntu/cephtest 2024-03-18T14:44:52.777 INFO:teuthology.orchestra.run.smithi092.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 14:44 /home/ubuntu/cephtest 2024-03-18T14:44:52.778 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T14:44:52.788 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T14:44:52.809 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T14:44:52.832 DEBUG:teuthology.nuke:shortname: smithi006 2024-03-18T14:44:52.832 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T14:44:52.857 DEBUG:teuthology.nuke:shortname: smithi092 2024-03-18T14:44:52.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T14:44:52.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:18.842886', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQtYpzYGP9RtuDnRfRAjMIFbIwY8l0IVJ419joOtQa91wnFCza3eQSAAxQj5WQX7MEHuvT3LFPqwb5o4+jEM8M='} 2024-03-18T14:44:52.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608831', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:21:18.840893', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1btTUr4MtuN4TIKkNYtJ8gv15N6zwZK5VqgoONnMI+bye+UznISZRSsJ0R/IXGCamLgfoAwgbJ30P0fFhjZ3g='} 2024-03-18T14:44:52.909 INFO:teuthology.orchestra.console.smithi006:Power off 2024-03-18T14:44:52.910 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T14:44:52.932 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-03-18T14:44:52.932 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:44:52.950 INFO:teuthology.orchestra.console.smithi092:Power off 2024-03-18T14:44:52.950 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T14:44:52.960 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-18T14:44:52.974 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-03-18T14:44:52.974 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:44:52.996 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-18T14:44:56.961 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:44:56.997 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:44:57.072 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-18T14:44:57.106 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-18T14:45:01.078 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:45:01.106 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:45:01.189 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-18T14:45:01.218 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-18T14:45:05.193 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:45:05.219 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T14:45:05.304 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-03-18T14:45:05.328 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-03-18T14:45:05.405 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-03-18T14:45:05.429 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-03-18T14:45:05.534 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} duration: 685.8875806331635 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=907dc3ee24c34b318112b65a654e49b0 status: fail success: false 2024-03-18T14:45:05.535 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T14:45:05.623 INFO:teuthology.run:FAIL